Content generation method, content generation model training method, device, electronic equipment and storage medium

Abstract

The present disclosure provides a content generation method, a training method and device of a content generation model, an apparatus and a storage medium, relates to the technical field of computers, and in particular to the fields of artificial intelligence, deep learning, model training, computer vision, speech technology and the like. The method comprises: based on text data, generating an audio sequence and a video sequence in a streaming manner by performing the following operations on a block-by-block basis, the text data describing to-be-generated audio content and to-be-generated video content: performing a self-attention operation based on a query vector of video noise of a to-be-generated video block, a query vector of audio noise of a to-be-generated audio block, an extended key vector and an extended value vector, updating the video noise and the audio noise, the extended key vector comprising a key vector of the video noise, the audio noise, historical video, historical audio and a text block, the extended value vector comprising a value vector of the video noise, the audio noise, the historical video, the historical audio and the text block, the text block being part or all of the text data.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, deep learning, model training, computer vision, and speech technology. More specifically, this disclosure provides a content generation method, a training method for a content generation model, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] With the development of deep learning generative models, audio and video generation technology has been widely applied in fields such as virtual interaction, film and television production, and online entertainment. These application scenarios place higher demands on generation quality, such as the continuity of generated content and multimodal synchronization performance. In audio and video generation, some solutions employ a diffusion transformer architecture, which gradually recovers the target data from random noise through a multi-step denoising process, generating short clips with clear visuals and timing matching the audio. Other solutions adopt a separate generation architecture, where audio and video are generated by independent models or channels, each optimizing its own modal generation quality, and finally performing timing alignment or splicing at the output. Summary of the Invention

[0003] This disclosure provides a content generation method, a training method for a content generation model, an apparatus, an electronic device, and a storage medium.

[0004] According to one aspect of this disclosure, a content generation method is provided, the method comprising: acquiring text data, the text data describing audio content to be generated and video content to be generated; based on the text data, performing the following operations block by block in a streaming manner to generate an audio sequence and a video sequence: performing a self-attention operation based on a query vector of video noise of a video block to be generated, a query vector of audio noise of an audio block to be generated, an extended key vector, and an extended value vector to update the video noise and audio noise, wherein the extended key vector includes the key vectors of video noise, audio noise, historical video, historical audio, and text blocks, and the extended value vector includes the value vectors of video noise, audio noise, historical video, historical audio, and text blocks, and the text blocks are part or all of the text data.

[0005] According to another aspect of this disclosure, a training method for a content generation model is provided. The method includes: acquiring sample text, which describes sample audio content and sample video content to be generated; inputting the sample text into an initial content generation model, and performing the following operations block by block using the initial content generation model to obtain sample audio sequences and sample video sequences: performing a self-attention operation based on the query vector of sample video noise for the sample video block to be generated, the query vector of sample audio noise for the sample audio block to be generated, the sample extended key vector, and the sample extended value vector, updating the sample video noise and sample audio noise, wherein the sample extended key vector includes the key vectors of sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text blocks, and the sample extended value vector includes the value vectors of sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text blocks, and the sample text block is part or all of the sample text; determining a loss based on the differences between the sample audio sequence, sample video sequence, and target audio sequence and target video sequence; and updating the parameters of the initial content generation model based on the loss to obtain a target content generation model.

[0006] According to another aspect of this disclosure, a content generation apparatus is provided, the apparatus including an acquisition module for acquiring text data, the text data describing audio content to be generated and video content to be generated; and a generation module for generating an audio sequence and a video sequence by performing the following operations block by block in a streaming manner based on the text data: performing a self-attention operation based on a query vector of video noise of the video block to be generated, a query vector of audio noise of the audio block to be generated, an extended key vector, and an extended value vector to update the video noise and audio noise, wherein the extended key vector includes the key vectors of video noise, audio noise, historical video, historical audio, and text blocks, and the extended value vector includes the value vectors of video noise, audio noise, historical video, historical audio, and text blocks, and the text blocks are part or all of the text data.

[0007] According to another aspect of this disclosure, a training apparatus for a content generation model is provided. The apparatus includes: an acquisition module for acquiring sample text, the sample text describing sample audio content and sample video content to be generated; a generation module for inputting the sample text into an initial content generation model, and using the initial content generation model to perform the following operations block by block to obtain a sample audio sequence and a sample video sequence: performing a self-attention operation based on the query vector of sample video noise and the query vector, sample extended key vector, and sample extended value vector of sample audio noise for the sample video block to be generated, updating the sample video noise and sample audio noise, wherein the sample extended key vector includes the key vectors of sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text blocks, and the sample extended value vector includes the value vectors of sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text blocks, and the sample text block is part or all of the sample text; a loss determination module for determining a loss based on the differences between the sample audio sequence, sample video sequence, and target audio sequence and target video sequence; and an update module for updating the parameters of the initial content generation model based on the loss to obtain a target content generation model.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the methods described above.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 The illustration schematically shows an exemplary system architecture of a content generation method, a content generation model training method, and an apparatus according to embodiments of the present disclosure;

[0014] Figure 2 A flowchart illustrating a content generation method according to an embodiment of the present disclosure is shown schematically;

[0015] Figure 3 A flowchart illustrating a self-attention operation according to an embodiment of the present disclosure is shown schematically;

[0016] Figure 4 A flowchart illustrating a training method for a content generation model according to an embodiment of the present disclosure is shown schematically;

[0017] Figure 5 The illustration shows a schematic diagram of a training process according to an embodiment of the present disclosure;

[0018] Figure 6 The illustration shows a schematic diagram of an application process according to an embodiment of the present disclosure;

[0019] Figure 7 A block diagram of a content generation apparatus according to an embodiment of the present disclosure is shown schematically;

[0020] Figure 8 A block diagram illustrating a training apparatus for a content generation model according to embodiments of the present disclosure is shown schematically; and

[0021] Figure 9 A block diagram of an electronic device suitable for implementing a content generation method and a training method for a content generation model according to embodiments of the present disclosure is shown schematically. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] As used in the specification and appended claims of this disclosure, the singular expressions “a,” “an,” and “the” are intended to include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this disclosure, “at least one,” “at least one,” and “one or more” refer to one, two, or more than two. “First,” “second,” and various numerical designations are merely distinctions for ease of description and are not intended to limit the scope of the embodiments of this disclosure. “And / or” describes the correspondence between corresponding objects, indicating that three relationships can exist. For example, “A and / or B” can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship. The sequence numbers of the processes below do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. For example, in the embodiments of this disclosure, the words “301,” “401,” and “501” are merely identifiers for ease of description and do not limit the order of execution steps.

[0024] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this disclosure include a specific feature, structure, or characteristic described in connection with that embodiment. In this disclosure, words such as "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this disclosure should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. In embodiments of this disclosure, descriptions such as "when," "in the case of," "if," and "if" indicate that the device will perform a corresponding action under certain objective circumstances, not a limitation on time, and do not require the device to necessarily perform a judgment action during implementation, nor do they imply any other limitations. In this disclosure, "for indicating" can include both direct and indirect indication. When describing an indication message as indicating A, it can include whether the indication message directly indicates A or indirectly indicates A, but does not necessarily mean that the indication message carries A.

[0025] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of any type of information, such as user personal information, comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0026] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0027] In the field of audio and video generation, joint audio and video generation methods employ a diffusion transformer architecture, which gradually recovers the target data from random noise through a multi-step denoising process, generating short clips with clear visuals and timing matching the audio. Other solutions adopt a separate generation architecture, where audio and video are generated by independent models or channels, each optimizing the generation quality of its modality, and finally performing timing alignment or splicing at the output.

[0028] However, all of the above solutions have certain problems and are difficult to meet practical needs, especially the requirements for real-time audio and video synchronization. Diffusion-based methods based on multi-step denoising iterations require multiple progressive refinement calculations for each generated segment during the inference process, resulting in generation time far exceeding the real-time playback rate, making it difficult to meet the low-latency requirements of real-time output. In separate audio and video generation schemes, audio and video are generated independently by different models, lacking a frame-level alignment mechanism, which may lead to problems such as poor lip print synchronization and mismatch between sound effects and visuals.

[0029] In view of this, embodiments of the present disclosure provide a scheme that generates audio and video segments in a streaming manner, that is, synchronously generates audio sequences and video sequences. During the generation of each audio sequence and video sequence, audio tokens and video tokens are jointly modeled in the same self-attention, so that audio tokens and video tokens interact directly in the same attention matrix, realizing frame-level audio and video synchronization information transmission, which helps to ensure the synchronization of real-time generated audio and video content.

[0030] Figure 1 This is a schematic diagram of an exemplary system architecture for methods and apparatus applicable to embodiments of this disclosure, based on one embodiment of the present disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0031] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0033] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0034] A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. A server can also be a server for a distributed system, or a server that incorporates blockchain technology.

[0035] It should be noted that the content generation method and the content generation model training method provided in this embodiment can be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the content generation apparatus and the content generation model training apparatus provided in this embodiment can also be disposed in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0036] Alternatively, the content generation method and the content generation model training method provided in this embodiment of the disclosure can be executed by the server 105. Accordingly, the content generation apparatus and the content generation model training apparatus provided in this embodiment of the disclosure can generally be located in the server 105.

[0037] Alternatively, the content generation method provided in this embodiment can be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103, and the training method for the content generation model can be executed by the server 105. Accordingly, the content generation apparatus provided in this embodiment can be disposed in the first terminal device 101, the second terminal device 102, and the third terminal device 103, and the training apparatus for the content generation model can be disposed in the server 105.

[0038] Alternatively, the content generation method and the content generation model training method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and is capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the content generation apparatus and the content generation model training apparatus provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and is capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] Figure 2 This is a flowchart of a content generation method according to an embodiment of the present disclosure. Figure 2 The method shown can be executed by a server, by a terminal device, or by both a server and a terminal device. For example, Figure 2 The method 200 shown can be executed by the system architecture described above.

[0041] like Figure 2 As shown, method 200 may include the following steps.

[0042] Step S210: Obtain text data, which describes the audio content and video content to be generated.

[0043] Step S220: Based on the text data, perform the following operations in a streaming, chunk-by-chunk manner to generate audio and video sequences:

[0044] A self-attention operation is performed based on the query vector of video noise for the video block to be generated, the query vector of audio noise for the audio block to be generated, the extended key vector, and the extended value vector to update the video noise and audio noise. The extended key vector includes the key vectors of video noise, audio noise, historical video, historical audio, and text blocks, and the extended value vector includes the value vectors of video noise, audio noise, historical video, historical audio, and text blocks. The text blocks are part or all of the text data.

[0045] In method 200, audio and video are generated in a streaming, chunk-by-chunk manner using a content generation model. That is, the content generation model generates one chunk at a time, and the generation process of each chunk depends on the chunks previously generated by the model. The final audio and video sequences are determined based on the chunks generated each time. For ease of description, in method 200, the content generation model is simply referred to as the model.

[0046] "chunk" can also be replaced with other descriptions such as "fragment".

[0047] Each chunk consists of an audio block and a video block of corresponding duration. A video block may include one or more video frames. An audio block may include one or more audio frames. It should be understood that audio frames and video frames are not in a one-to-one correspondence; a video block may include one or more video frames, and the audio block of the corresponding duration is the audio corresponding to those one or more video frames.

[0048] Alternatively, it can be understood that each chunk includes audio tokens and video tokens of corresponding duration. A video chunk can include video tokens for one or more video frames. An audio chunk can include audio tokens for one or more audio frames.

[0049] The following is an example of a streaming, chunk-by-chunk generation process:

[0050] Step 1: Generate the first chunk x_1.

[0051] x_1=G(c_text,noise_1).

[0052] Step 2: Generate the second chunk x_2.

[0053] x_2=G(c_text,x_1,noise_2).

[0054] Step 3: Generate the third chunk x_3.

[0055] x_3=G(c_text,x_1,x_2,noise_3).

[0056] …

[0057] Here, x_1 includes the first audio block x_1_a and the first video block x_1_v, and so on. c_text represents the text condition. noise_1 includes audio noise noise_1_a and video noise noise_1_v, and so on. G() represents the content generation model. In the above generation process, each chunk to be generated depends on the chunks previously generated by the model (i.e., historical chunks), or in other words, the model performs autoregressive generation conditioned on its own generated output.

[0058] It should be understood that the above is merely an example of a generation process and does not constitute a limitation on the implementation of this disclosure. Other generation processes can be found in the descriptions below.

[0059] Text data can be text prompts.

[0060] The text data may also include one or more text blocks.

[0061] In the case where the text data includes a text block, that text block is the text data itself.

[0062] In the process of generating audio and video chunk by chunk, the text prompt input to the model can be the same for each chunk. That is, the model input includes this prompt each time.

[0063] When the text data consists of multiple text blocks, it can be understood as a sequence of prompts, with each text block considered as a prompt. For different chunk generation processes, the text conditions input to the model can be the same or different text blocks (prompts).

[0064] Text blocks and chunks can have a one-to-one correspondence. For example, the text data can be a sequence of prompts divided into chunks. For the generation process of each chunk, the text conditions input to the model are the prompts corresponding to that chunk.

[0065] Alternatively, the correspondence between text blocks and chunks may not be one-to-one. For example, a text block can correspond to multiple chunks, and correspondingly, the text conditions input to the model for the generation of these multiple chunks are all based on that text block.

[0066] As an example, text data can be generated based on audio / video generation requests.

[0067] For example, the text data may be generated by a large language model (LLM) based on an audio / video generation request.

[0068] For example, an audio / video generation request could be "Generate a 5-minute educational video about the solar system." The LLM generates a chunked prompt sequence based on this request, which is the text data in step S210.

[0069] The above are just examples. Text data can also be generated in other ways, such as user input. This disclosure does not limit this.

[0070] This text data can be acquired in a single step. It simultaneously describes both the visual and audio content to be generated, thus helping to ensure semantic consistency between audio and video.

[0071] Alternatively, the text data can be obtained multiple times; for example, the text describing the audio content to be generated and the text describing the video content to be generated can be obtained separately.

[0072] In method 200, the specific structure of the model can be selected as needed.

[0073] For example, the model can use a multi-modal diffusion transformer (MMDiT) as the backbone network. Alternatively, the model can also use other Transformer architectures that support unified multimodal modeling, such as encoding audio and video tokens separately and then interacting with them through interleaved attention, as long as the architecture includes a self-attention layer. This disclosure does not limit the specific structure of the model.

[0074] The model may include one or more self-attention layers. For example, the model may include one or more transformer layers, each of which includes a self-attention layer.

[0075] Step S220 will be explained below.

[0076] The chunk to be generated includes the video chunk and the audio chunk. The chunk to be generated can also be called the current chunk.

[0077] For a self-attention layer, the input can include video noise for the video block to be generated and audio noise for the audio block to be generated. The input can also include text blocks. The output of this self-attention layer is used to update the video noise and audio noise.

[0078] The specific format of the input data can be set as needed.

[0079] The video noise input to the self-attention layer for the video block to be generated can be the video tokens of the video noise for the video block to be generated.

[0080] The audio noise input to the self-attention layer for the audio block to be generated can be the audio tokens of the audio noise of the audio block to be generated.

[0081] The text blocks input to the self-attention layer can be text tokens of the text blocks.

[0082] For example, video tokens and audio tokens can be concatenated into a unified sequence and input into the self-attention layer. Further, video tokens, audio tokens, and text tokens can be concatenated into a unified sequence and input into the self-attention layer.

[0083] Perform a multi-head self-attention mechanism on the data input to the self-attention layer.

[0084] The process of attention operations is explained below. In a self-attention layer, the following steps can be performed.

[0085] (1) Calculate the query (Q) vector, key (K) vector, and value (V) vector of the video noise of the video block to be generated and the audio noise of the audio block to be generated.

[0086] (2) Obtain the K vector of historical video, the K vector of historical audio and the K vector of text block.

[0087] (3) Obtain the V vector of historical video, the V vector of historical audio and the V vector of text block.

[0088] Historical video can be understood as videos (or video tokens) generated by the model before the video blocks to be generated. Historical audio can be understood as audio (or audio tokens) generated by the model before the audio blocks to be generated.

[0089] The historical video and historical audio used for attention calculation can be corresponding. For example, the historical video can be one or more video frames (or video tokens of one or more video frames) generated before the video block to be generated. The historical audio can be understood as the audio (or audio tokens) generated before the audio block to be generated that corresponds to the one or more video frames.

[0090] The K-vector of a historical video can be understood as the K-vector of the historical video at that layer, and the V-vector is similar.

[0091] The K vector of historical audio can be understood as the K vector of historical audio at that layer, and the V vector is similar.

[0092] The K vector of a text block can be understood as the K vector of the text block at that layer, and the V vector is similar.

[0093] For ease of description, the K vector and V vector are collectively referred to as KV vectors.

[0094] The "get" in steps (2) and (3) can be implemented in several ways. For example, "get" can be "compute". For example, "get" can be "read from cache". For example, after each generation of video and audio blocks, the key-value pairs of the newly generated audio and video blocks can be stored in the cache for use in the generation of subsequent chunks.

[0095] The extended K-vector includes the K-vector of video noise, the K-vector of audio noise, the K-vector of historical video, the K-vector of historical audio, and the K-vector of text blocks. For example, the extended K-vector can be obtained by concatenating the above keys.

[0096] The extended V vector includes the V vectors of video noise, audio noise, historical video, historical audio, and text blocks. For example, the extended V vector can be obtained by concatenating the above values.

[0097] (4) Perform self-attention operation based on the Q vector, extended K vector and extended V vector of video noise and audio noise.

[0098] The self-attention operation in step S220 is performed in a self-attention layer. The operation can be performed in a similar way for each self-attention layer.

[0099] Figure 3 This diagram illustrates a portion of the chunk generation process. For example... Figure 3 As shown, the self-attention operation can be performed in each Transformer layer in the manner described above.

[0100] For example, such as Figure 3 As shown, in each Transformer layer, all tokens are concatenated into a unified sequence X:

[0101] X = [c_text; x_a; x_v];

[0102] c_text represents text block tokens, x_a represents audio tokens, and x_v represents video tokens.

[0103] Applying standard multi-head self-attention to this unified sequence X can also be understood as employing a unified self-attention mechanism:

[0104] Attention(Q, K, V) = softmax(QK^T / sqrt(d)) V;

[0105] Here, `Attention()` represents the self-attention mechanism, and `Q`, `K`, and `V` represent the query vector, key vector, and value vector, respectively. `Q = X*W_Q`, `K = X*W_K`, `V = X*W_V`. `W_Q` represents the query weight matrix, `W_K` represents the key weight matrix, and `W_V` represents the value weight matrix. `D` represents the dimension of each attention head.

[0106] According to the embodiments of this disclosure, audio and video are generated chunk-by-chunk in a streaming manner, which facilitates real-time audio and video generation and enables interactive audio and video generation. For example, during the real-time generation of audio and video, if a new user input request is received, a text prompt can be determined based on the new request and used as the new prompt for continued output. Simultaneously, audio tokens and video tokens can interact directly within the same attention matrix. Audio tokens can be fused with video information, and video tokens can be fused with audio information, achieving frame-level audio and video synchronization information transmission. This helps ensure the synchronization of real-time generated audio and video content; for example, lip-sync and sound effect matching quality are significantly better than in separate solutions. Furthermore, text conditions also participate in self-attention, eliminating the need for an additional cross-attention module.

[0107] Meanwhile, this solution can generate audio and video from the same model simultaneously, eliminating the need to maintain multiple independent models, which helps reduce deployment and maintenance costs.

[0108] The following is an explanation of the historical videos and audio recordings.

[0109] In some embodiments, historical video may include video within a first time period at the beginning of a video sequence, and historical audio may include audio within a first time period at the beginning of an audio sequence.

[0110] The word "first" in "first period" is used for descriptive convenience only and has no limiting effect.

[0111] In other words, during the generation of each chunk, the key-value vectors of the video and audio tokens generated at the beginning of the process need to be used when performing the self-attention operation. The video and audio tokens during this period can be regarded as frame sinks.

[0112] As one possible implementation, the historical video may include S video frames at the beginning of the video sequence, where S is a positive integer, and the historical audio may include audio from the audio sequence corresponding to the S video frames.

[0113] The audio corresponding to S video frames can be understood as the audio segment and the S video frames corresponding to the same time period.

[0114] In other words, during the generation of each chunk, the key-value vectors of the video tokens and corresponding audio tokens from the initially generated S video frames are required when performing the self-attention operation. These S video tokens and corresponding audio tokens can be considered as frame sinks.

[0115] For example, historical video and historical audio may include the first P chunks generated by the model, where P is a positive integer. These P chunks include video tokens and corresponding audio tokens for S video frames. For instance, if P is 1, the historical video and historical audio include the first chunk generated by the model. This first chunk may include video tokens and corresponding audio tokens for S video frames.

[0116] Alternatively, historical video and historical audio may include partial video frames and corresponding audio content from the first P chunks generated by the model. For example, if P is 1, then historical video and historical audio may include video tokens and corresponding audio tokens from partial video frames in the first chunk generated by the model.

[0117] As an example, the key-value vectors of the video tokens and corresponding audio tokens of the S video frames can always be kept in the key-value cache and concatenated to the key and value in each subsequent self-attention calculation.

[0118] According to the embodiments of this disclosure, the key-value vectors of the first generated video and audio content are used for the generation of subsequent audio and video content, providing a globally consistent anchor point. For example, it can provide consistent identity information, scene layout, and overall style reference for all subsequent chunks, preventing identity drift and style degradation during long-term generation, thereby further ensuring the stability of generation quality.

[0119] In some embodiments, historical video may further include video from a second time period adjacent to the video block to be generated in the video sequence preceding the video block to be generated, and historical audio may further include audio from a second time period adjacent to the audio block to be generated in the audio sequence preceding the audio block to be generated.

[0120] The "second" in "second period" is for descriptive convenience only and has no limiting effect.

[0121] In other words, during the generation of each chunk, the key-value vectors of video and audio tokens generated by the model in the most recent time period are required when performing the self-attention operation. The second time period is always the time period adjacent to the current chunk. When a new chunk is generated, the second time period changes and scrolls forward; the second time period can also be called the rolling window. The length of the second time period is the length of the rolling window.

[0122] As one possible implementation, the historical video also includes W video frames adjacent to the video block to be generated in the video sequence, where W is a positive integer, and the historical audio also includes audio in the audio sequence corresponding to the W video frames.

[0123] In other words, during the generation of each chunk, when performing the self-attention operation, the key-value vectors of the video tokens and corresponding audio tokens of the W most recently generated video frames are required. At this point, the key-value vectors of the video tokens and corresponding audio tokens of these W video frames are the key-value vectors within the rolling window.

[0124] For example, historical video and historical audio can be included in the M most recently generated chunks of the model. M is a positive integer. The M chunks include video tokens and corresponding audio tokens for W video frames. For example, if M is 2, then historical video and historical audio include the two most recently generated chunks of the model. These two chunks can include video tokens and corresponding audio tokens for W video frames.

[0125] Alternatively, historical video and historical audio can include partial video frames and corresponding audio content from the M most recently generated chunks by the model. For example, M is 2, meaning historical video and historical audio can include video tokens and corresponding audio tokens from partial video frames in two chunks generated by the model.

[0126] As an example, a key-value cache (KV cache) is maintained for the video tokens and corresponding audio tokens of the most recent W video frames, using a first-in-first-out (FIFO) strategy. This means that when new video or audio content is generated, the KV cache containing the video tokens and corresponding audio tokens of those W video frames is evicted. This ensures that the KV cache always retains the KV vectors of the most recently generated W video frames' video tokens and corresponding audio tokens, which are then concatenated into the Keys and Values ​​during subsequent self-attention calculations.

[0127] According to the embodiments of this disclosure, the key-value vectors of the most recently generated video and audio content are used for the generation of subsequent audio and video content, providing a local temporal context, which helps to ensure the motion coherence between adjacent frames, that is, to ensure smooth motion between adjacent frames.

[0128] The following example illustrates how historical video and audio can originate from a frame sink and a rolling window.

[0129] For example, the calculation of attention at each step can be referenced from the following formula:

[0130] Attention(Q_current, [K_sink; K_rolling], [V_sink; V_rolling]);

[0131] In this scheme, `Attention()` represents the attention calculation, `Q_current` represents the Q-vector of the current chunk (i.e., the Q-vector of the video noise of the video chunk to be generated and the audio noise of the audio chunk to be generated), `[K_sink; K_rolling]` represents the K-vector of the historical video and audio, and `[V_sink; V_rolling]` represents the V-vector of the historical video and audio. `K_sink` and `V_sink` represent the KV vectors of the first S video frames and their corresponding audio, with fixed sizes and never being discarded. `K_rolling` and `V_rolling` represent the KV vectors of the most recent W video frames and their corresponding audio, updated in a FIFO (First-In, First-Out) manner. In this scheme, subsequent chunks are not generated based on all previously generated video and audio content, but rather on the first S generated video frames and their corresponding audio, and the most recent W generated video frames and their corresponding audio. The total KV cache size is O(S + W), and it does not increase with the total generation duration. Therefore, this scheme can support audio and video generation of unlimited duration.

[0132] It should be understood that the above formula only shows the key-value vectors of historical video and historical audio, and does not limit the attention calculation to only the key-value vectors in the above formula. Attention calculation can also involve other key-value vectors, and a detailed description can be found in the previous text.

[0133] The following example illustrates the above solution.

[0134] Assume that the historical video and historical audio include the first chunk generated by the model and the most recently generated chunk.

[0135] As mentioned earlier, the model generates audio and video chunk by chunk in a streaming manner. The model's input includes the text prompt (c_text).

[0136] Initialization phase: The first chunk is generated, which serves as the frame sink. The process of generating this chunk can be referred to in the previous text and will not be repeated here.

[0137] Looping phase: Repeat the following steps until termination, such as when the user stops or content generation ends.

[0138] a. Retrieve the context from the current KV Cache.

[0139] The KV cache includes the KV vectors of the Frame Sink and Rolling Window. In this example, the KV cache includes the KV vectors of the first chunk and the KV vectors of the most recently generated chunk.

[0140] b. Forward reasoning to generate the current chunk.

[0141] This generates audio tokens and video tokens for the current chunk.

[0142] c. Add the KV vector of the current chunk to the KV cache, and use FIFO to evict the KV cache of the oldest chunk.

[0143] d. Output the audio and video of the current chunk.

[0144] Assuming we are now executing step 6, generating the 6th chunk x_6 (i.e., the chunk to be generated). The historical video and audio include the first chunk generated by the model (i.e., x_1) and the most recently generated chunk (i.e., x_5).

[0145] Accordingly, x_6 = G(c_text, x_1, x_5, noise_6).

[0146] For the Q vector: compute the Q vector for each token in noise_6_a (i.e., audio noise) and noise_6_v (i.e., video noise).

[0147] For KV vectors: Calculate the KV vector for each of the following items:

[0148] noise_6_a and noise_6_v;

[0149] c_text;

[0150] x_1_a and x_1_v;

[0151] x_5_a and x_5_v.

[0152] Self-attention calculation is performed based on the Q-vector and KV-vector calculated above. The result of the self-attention calculation is used to update noise_6_a and noise_6_v.

[0153] According to the embodiments of this disclosure, the video and audio KV vectors generated in the earliest period and the video and audio KV vectors generated in the most recent period are retained for use in the generation of subsequent audio and video content. The overall size of the KVCache remains unchanged, the memory usage is fixed and does not increase with the total generation time, and it can support the continuous generation of audio and video of any duration (including unlimited duration) without being limited by storage space.

[0154] It should be understood that the above is only an example of one possible implementation. For example, in other implementations, a memory bank or learnable register tokens can also be used as global anchors.

[0155] In some embodiments, the video block and audio block to be generated are generated through single-step inference.

[0156] In other words, each chunk can be generated with only one forward inference step, without the need for multi-step denoising. This model is a single-step inference model.

[0157] According to the embodiments of this disclosure, the video block and audio block to be generated are generated through single-step inference, without the need for multi-step noise reduction, which significantly improves the generation speed and helps to ensure that the generation speed reaches the real-time or super real-time level, thereby helping to ensure the generation effect in interactive scenes.

[0158] The content generation model in Method 200 can be trained using conventional training methods, such as teacher forcing. However, this method uses ground truth frames as conditions during training, but can only use frames generated by the model itself (which may have errors) as conditions during inference, leading to error accumulation.

[0159] In view of this, this disclosure also provides a training method for a content generation model, which adopts a self-forcing training strategy. During training, the model uses its own generated output as a condition for autoregressive generation to solve the problem of training-inference distribution mismatch, thereby improving the generation quality of the model.

[0160] In some embodiments, the content generation model is obtained by updating the parameters of an initial content generation model based on a loss determined by the differences between sample audio sequences, sample video sequences, and target audio sequences and target video sequences. The sample audio sequences and sample video sequences are obtained by inputting sample text into the initial content generation model and performing the following operations block by block using the initial content generation model:

[0161] A self-attention operation is performed based on the query vector, sample expansion key vector, and sample expansion value vector of the sample video noise and the sample audio noise of the sample audio block to be generated, to update the sample video noise and sample audio noise. The sample expansion key vector includes the key vectors of the sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text block. The sample expansion value vector includes the value vectors of the sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text block. The sample text block is part or all of the sample text, which describes the sample audio content and sample video content to be generated.

[0162] For specific training procedures, please refer to Method 400 below.

[0163] Figure 4 A schematic flowchart illustrating a training method for a content generation model according to an embodiment of the present disclosure is shown.

[0164] The content generation model in method 200 can be achieved through... Figure 4 The target content generation model trained by the method 400 shown can be used as the content generation model in the method 200.

[0165] Figure 4 The method shown can be executed by a server, by a terminal device, or by both a server and a terminal device. For example, Figure 4 The method 400 shown can be executed by the system architecture described above.

[0166] like Figure 4 As shown, method 400 may include the following steps.

[0167] Step S410: Obtain sample text. The sample text describes the audio content and video content to be generated.

[0168] Step S420: Input the sample text into the initial content generation model, and use the initial content generation model to perform the following operations block by block in a streaming manner to obtain the sample audio sequence and sample video sequence:

[0169] A self-attention operation is performed based on the query vector of the sample video noise of the sample video block to be generated and the query vector of the sample audio noise of the sample audio block to be generated, the sample expansion key vector and the sample expansion value vector to update the sample video noise and sample audio noise. The sample expansion key vector includes the key vectors of the sample video noise, sample audio noise, sample historical video, sample historical audio and sample text block, and the sample expansion value vector includes the value vectors of the sample video noise, sample audio noise, sample historical video, sample historical audio and sample text block, where the sample text block is part or all of the sample text.

[0170] Step S430: Determine the loss based on the differences between the sample audio sequence, the sample video sequence, the target audio sequence, and the target video sequence.

[0171] Step S440: Update the parameters of the initial content generation model based on the loss to obtain the target content generation model.

[0172] In method 400, the model performs an autoregressive rollout using its own generated output as a condition. The model generates content prior to the sample video and audio chunks to be generated, using historical sample videos and audio as initial content. That is, the generation process of each sample chunk depends on the sample chunks previously generated by the model, and the final generated sample audio and video sequences are determined based on each generated sample chunk.

[0173] The specific process of forward propagation can be consistent with the reasoning process of Method 200 mentioned above. The descriptions of sample text, sample audio sequences, sample video sequences, sample video blocks to be generated, sample video noise, sample audio blocks to be generated, sample audio noise, sample extended key vectors, sample extended value vectors, sample historical videos, sample historical audio, and sample text blocks can be found in the specific descriptions of text data, audio sequences, video sequences, video blocks to be generated, video noise, audio blocks to be generated, audio noise, extended key vectors, extended value vectors, historical videos, historical audio, and text blocks in Method 200, and will not be repeated here.

[0174] For example, incremental computation can be performed using a KV Cache during training.

[0175] In some embodiments, the sample history video may include video within a first time period at the start of the sample video sequence, and the sample history audio may include audio within a first time period at the start of the sample audio sequence.

[0176] As one possible implementation, the sample history video may include S sample video frames at the start of the sample video sequence, where S is a positive integer, and the sample history audio may include sample audio corresponding to the S sample video frames in the sample audio sequence.

[0177] In some embodiments, the sample history video may further include video from a second time period adjacent to the sample video block to be generated in the sample video sequence, and the sample history audio may further include audio from a second time period adjacent to the sample audio block to be generated in the sample audio sequence.

[0178] As one possible implementation, the sample history video may also include W sample video frames adjacent to the sample video block to be generated in the sample video sequence, where W is a positive integer, and the sample history audio may also include sample audio corresponding to the W sample video frames in the sample audio sequence.

[0179] For a detailed description of the above process, please refer to Method 200 above. For example, the relevant descriptions of the first and second time periods can be found in Method 200 above. To avoid repetition, they will not be repeated here.

[0180] According to the scheme of this disclosure embodiment, during training, the model is allowed to perform autoregressive generation using its own generated output as a condition, rather than using the ground truth as a condition, to avoid training-inference distribution mismatch, thereby improving the generation effect of the model.

[0181] As one possible implementation, the differences between the sample audio sequence, sample video sequence and the target audio sequence, target video sequence can be the differences between the distributions of the sample audio sequence, sample video sequence and the distributions of the target audio sequence, target video sequence.

[0182] The distributions of the sample audio and video sequences can be understood as the output distributions of the initial content generation model. Accordingly, the training objective can be to minimize the difference between the output distribution of the initial content generation model and the distributions of the target audio and video sequences.

[0183] In this way, by taking the reduction of the difference between the model's output distribution and the target distribution as the training objective, the optimization targets the overall sequence quality rather than frame-by-frame reconstruction, which is more suitable for streaming generation scenarios.

[0184] In some embodiments, the differences between the sample audio sequence, the sample video sequence, and the target audio sequence, the target video sequence are determined based on the divergence between the distributions of the sample audio sequence, the sample video sequence, and the distributions of the target audio sequence, the target video sequence.

[0185] Step S440 may include: updating the parameters of the initial content generation model based on the loss through model distillation, wherein the initial content generation model is a student model, and the target audio sequence and target video sequence are obtained by inputting sample text into the teacher model.

[0186] The sample text is the training text prompt. The training text prompt is input into the teacher model, which generates the target audio sequence and the target video sequence.

[0187] In this case, the distribution of the target audio sequence and the target video sequence can be understood as the output distribution of the teacher model. Accordingly, the training objective can be understood as minimizing the divergence between the output distribution of the student model and the output distribution of the teacher model, for example, using distribution matching distillation (DMD) as the training objective.

[0188] The output of the teacher model can serve as a supervisory signal during training, eliminating the need for manually labeled data. The teacher model can generate supervisory signals simply by using sample text (i.e., text prompts).

[0189] According to the scheme of this disclosure embodiment, the visual content and audio content to be generated are described simultaneously by sample text (i.e. training text prompt), which ensures the semantic consistency of audio and video. The teacher model is driven to generate supervision signals by sample text, eliminating the need for manually labeled paired audio and video datasets and greatly reducing the cost of obtaining training data.

[0190] In some embodiments, the teacher model is a multi-step denoising model, and the student model is a single-step inference model.

[0191] In other words, the teacher model is a multi-step generator, and the student model is a single-step generator.

[0192] For example, the teacher model can employ a pre-trained LTX-2 multi-step diffusion model.

[0193] For example, the student model can employ MMDiT.

[0194] For example, the model's loss can satisfy the following formula

[0195] L_DMD = D_KL(p_teacher(x_1:N | c_text) || p_student(x_1:N | c_text))

[0196] Where L_DMD represents the loss, and D_KL() represents the Kullback-Leibler (KL) divergence. p_teacher is the audio and video generation distribution of the teacher model after multi-step denoising, p_student is the audio and video generation distribution obtained by the student model in a single step of inference, x_1:N is the complete N-frame audio and video sequence, and c_text is the text condition (i.e., the training text prompt).

[0197] According to the embodiments of this disclosure, the initial content generation model is trained by distillation. The student model adopts a single-step inference model, and the teacher model is a multi-step denoising model. This is beneficial to distill the multi-step denoising capability of the teacher model into the single-step inference of the student model, so that the student model can achieve the generation quality of the teacher model's multi-step denoising in single-step generation.

[0198] The student model can learn the joint distribution of audio and video during the distillation process and gain real-time generation capabilities, which greatly improves the generation speed.

[0199] It should be understood that the above is only one possible implementation. In other implementations, other losses can be used as the training objective for distribution matching, such as score identity distillation (SiD) or generative adversarial network (GAN) loss.

[0200] In some embodiments, the sample text may be generated via LLM.

[0201] For example, LLM can be used to randomly generate training text prompts for preset scene categories (such as digital human dialogue, natural scenery, musical performance, news broadcast, animated characters, etc.). The generated prompts describe both the visual content and audio content to be generated, thus helping to ensure semantic consistency between audio and video.

[0202] In some embodiments, the sample text may be generated by a large model based on cue word templates for the target scene category.

[0203] Below is an example of automatically generating training text prompts using LLM.

[0204] (1) Multiple scene categories are preset, such as digital human dialogue, natural scenery, music performance, news broadcast, animated characters, etc. These multiple scene categories include the target scene category.

[0205] (2) Design a corresponding prompt template for each scene category. The prompt template includes a visual description (or video description) and an audio description.

[0206] (3) Use LLM to randomly generate specific prompts based on the prompt template of the target scene category, i.e. sample text prompts.

[0207] For example, during training, the generated sample text prompt is input into a pre-trained teacher model (such as LTX-2) to perform multi-step denoising to generate high-quality audio and video, and then used as a supervision signal to train the self-forcing autoregressive rollout of the student model (MMDiT).

[0208] According to the embodiments of this disclosure, LLM is used to generate sample text, which reduces the cost of generating sample text and improves the diversity of sample text, thereby helping to further ensure the training quality of the model.

[0209] It should be understood that the above is only one possible implementation method. Other methods can be used to generate sample text. For example, the sample text can be manually written. Another example is using manually annotated audio and video datasets in conjunction with a captioning model to generate training text prompts.

[0210] Figure 5 An example of a training process is shown. Figure 5 As shown, sample text prompts for each scene category are generated using LLM. These sample text prompts are then input into both the teacher and student models. The student model is trained using DMD based on the divergence between the output distributions of the teacher and student models. The teacher model is a multi-step denoising model, while the student model is a single-step inference model. The specific training process can be found in Method 300 above; to avoid repetition, it will not be elaborated here.

[0211] The student model employs a full self-attention mechanism, concatenating sample text tokens, sample video tokens, and sample audio tokens into a single sequence and modeling them within the same self-attention layer. Furthermore, the student model can utilize a KV Cache for incremental computation; that is, during self-attention computation, the KV vectors of historically generated content are derived from the KV Cache. The KV Cache includes the KV vectors of frame anchors (e.g., the first sample chunk) and the KV vectors of the scrolling window (e.g., the most recent W frames). Specifically, frame anchors are permanently stored in the KV Cache and concatenated into the Keys and Values ​​during attention computation in subsequent sample chunk generation. The KV vectors in the scrolling window are updated using a FIFO strategy. The specific forward propagation process can be found in Method 200 above; to avoid repetition, it will not be elaborated here.

[0212] The solutions of the embodiments of this disclosure can be applied to various real-time audio and video generation scenarios. For example, the content generation model of this disclosure can be used as a dedicated model for real-time long video generation.

[0213] For example, specific application scenarios may include popular science explanations, live knowledge broadcasts, news broadcasts, and visualization of audiobooks, which require long-term continuous audio and video output.

[0214] (1) Popular science explanation: The solution disclosed herein can generate popular science animations or live-action videos with synchronized audio narration in real time according to task requests (e.g., carrying popular science topics or scripts, etc.), supporting continuous output from several minutes to tens of minutes.

[0215] (2) Knowledge live streaming: The solution disclosed herein can be applied to the scenario of artificial intelligence (AI) lecturers on online education platforms, generating teaching videos and accompanying audio explanations in real time according to the course outline, and supporting continuous live streaming for an unlimited duration.

[0216] (3) News broadcasting: The disclosed solution can generate virtual anchor broadcasting video and synchronized voice in real time based on the press release text, which is suitable for 24-hour uninterrupted news channels.

[0217] (4) Visualization of audiobooks: The solution disclosed herein can convert the text of literary works into video streams with dubbing and visuals in real time for the visualization of long content.

[0218] It should be understood that the above are merely examples, and the solutions of the embodiments of this disclosure can also be applied to other real-time audio and video generation scenarios, which are not limited by this disclosure.

[0219] Below is an example of a specific application process.

[0220] (1) The user submits a generation request through the video agent.

[0221] For example, the generation request may include a task description (such as "Generate a 5-minute educational video about the solar system").

[0222] For example, the generation request may also include the context required for the task; for instance, in a news broadcasting scenario, the generation request may also include the news release text.

[0223] (2) The video agent routing module distributes tasks according to their type.

[0224] Specifically, if the video agent routing module identifies the request as a long video generation request and the task type as a long audio and video real-time generation task, then it will distribute the generation request to the content generation model in the embodiments of this disclosure.

[0225] (3) The LLM determines the structured prompt sequence of the chunk based on the generation request.

[0226] For example, if the request to generate includes press release text, LLM can break that press release text down into a structured sequence of chunks of prompts.

[0227] For example, if the generation request is "generate a 5-minute popular science video about the solar system", LLM can generate a prompt based on the generation request and decompose the prompt into a structured prompt sequence divided into chunks.

[0228] (4) Synchronized audio and video are generated in real time, chunk by chunk, through a content generation model.

[0229] The specific generation process can be found in Method 200 above, and will not be repeated here.

[0230] (5) Output the generated audio and video streams to the user terminal or live streaming platform or other designated locations in real time.

[0231] The above generation process continues until the content is completed or the user stops it voluntarily.

[0232] Figure 6 An example of an application process is shown. For example... Figure 6 As shown, LLM generates text prompts based on specific scenarios (such as science popularization, news broadcasts, and live knowledge sharing). The text prompts are then input into the content generation model, which generates synchronized audio and video in real-time, chunk by chunk, and outputs the audio and video streams in real time.

[0233] The content generation model employs a unified self-attention mechanism, concatenating text tokens, video tokens, and audio tokens into a single sequence and modeling them within the same self-attention layer. The content generation model can also utilize a KVCache for incremental computation; that is, during self-attention computation, the KV vectors of historically generated content are derived from the KVCache. The KVCache includes the KV vectors of frame anchors (e.g., the first sample chunk) and the KV vectors of the scrolling window (e.g., the most recent W frames). Specifically, frame anchors are permanently stored in the KVCache and concatenated into the Keys and Values ​​during attention computation in subsequent sample chunk generation. The KV vectors in the scrolling window are updated using a FIFO strategy. The specific process of forward propagation can be found in Method 200 above; to avoid repetition, it will not be elaborated here.

[0234] Figure 7 A block diagram of a content generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0235] like Figure 7 As shown, the content generation apparatus 700 may include an acquisition module 710 and a generation module 720.

[0236] The operations performed by modules 710 to 720 in the content generation device 700 and the effects they can achieve are similar to steps S210 to S220 in the content generation method described above. To avoid repetition, appropriate parts of the description will be used when describing the device 700.

[0237] The acquisition module 710 is used to acquire text data, which describes the audio content and video content to be generated.

[0238] The generation module 720 is used to generate audio and video sequences by performing the following operations block by block in a streaming manner based on text data: performing self-attention operations based on the query vector of video noise of the video block to be generated, the query vector of audio noise of the audio block to be generated, the extended key vector, and the extended value vector, and updating the video noise and audio noise. The extended key vector includes the key vectors of video noise, audio noise, historical video, historical audio, and text blocks, and the extended value vector includes the value vectors of video noise, audio noise, historical video, historical audio, and text blocks. The text blocks are part or all of the text data.

[0239] In some embodiments, historical video includes video within a first time period at the start of a video sequence, and historical audio includes audio within a first time period at the start of an audio sequence.

[0240] In some embodiments, the historical video also includes video from a second time period adjacent to the video block to be generated in the video sequence preceding the video block to be generated, and the historical audio also includes audio from a second time period adjacent to the audio block to be generated in the audio sequence preceding the audio block to be generated.

[0241] In some embodiments, the video block and audio block to be generated are generated through single-step inference.

[0242] In some embodiments, the video blocks and audio blocks to be generated are generated by a target content generation model. The target content generation model is obtained by updating the parameters of the initial content generation model based on a loss. The loss is determined based on the differences between the sample audio sequence, the sample video sequence, the target audio sequence, and the target video sequence. The sample audio sequence and sample video sequence are obtained by inputting sample text into the initial content generation model and performing the following operations block by block using the initial content generation model: performing a self-attention operation based on the query vector, sample expansion key vector, and sample expansion value vector of the sample video noise and the sample audio noise of the sample audio block to be generated, and updating the sample video noise and sample audio noise. The sample expansion key vector includes the key vectors of the sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text block. The sample expansion value vector includes the value vectors of the sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text block. The sample text block is part or all of the sample text, and the sample text describes the sample audio content and sample video content to be generated.

[0243] Figure 8 This is a block diagram of a training apparatus for generating a content model according to another embodiment of the present disclosure.

[0244] like Figure 8 As shown, the training device 800 for the content generation model may include an acquisition module 810, a generation module 820, a loss determination module 830, and an update module 840.

[0245] The operations performed by modules 810 to 840 in the training device 800 of the content generation model and the effects they can achieve are similar to steps S410 to S440 in the training method of the content generation model. To avoid repetition, appropriate parts of the description are provided when describing the device 800.

[0246] The acquisition module 810 is used to acquire sample text, which describes the audio and video content to be generated.

[0247] The generation module 820 is used to input sample text into the initial content generation model, and then perform the following operations block by block using the initial content generation model to obtain sample audio sequences and sample video sequences:

[0248] A self-attention operation is performed based on the query vector of the sample video noise of the sample video block to be generated and the query vector of the sample audio noise of the sample audio block to be generated, the sample expansion key vector, and the sample expansion value vector to update the sample video noise and sample audio noise. The sample expansion key vector includes the key vectors of the sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text block. The sample expansion value vector includes the value vectors of the sample video noise, sample audio noise, sample historical video, sample historical audio, and sample text block. The sample text block is part or all of the sample text.

[0249] The loss determination module 830 is used to determine the loss based on the differences between the sample audio sequence, the sample video sequence and the target audio sequence, and the target video sequence.

[0250] The update module 840 is used to update the parameters of the initial content generation model based on the loss, so as to obtain the target content generation model.

[0251] In some embodiments, the differences between the sample audio sequence, the sample video sequence, and the target audio sequence and the target video sequence are determined based on the divergence between the distributions of the sample audio sequence, the sample video sequence, and the distributions of the target audio sequence and the target video sequence. The update module 840 is specifically used to update the parameters of the initial content generation model based on the loss through model distillation. The initial content generation model is a student model, and the target audio sequence and the target video sequence are obtained by inputting sample text into the teacher model.

[0252] In some embodiments, the teacher model is a multi-step denoising model, and the student model is a single-step inference model.

[0253] In some embodiments, the sample text is generated using a large model.

[0254] In some embodiments, the sample text is generated by a large model based on cue word templates for the target scene category.

[0255] In some embodiments, the sample history video includes video within a first time period at the start of the sample video sequence, and the sample history audio includes audio within a first time period at the start of the sample audio sequence.

[0256] In some embodiments, the sample history video also includes video from a second time period adjacent to the sample video block to be generated in the sample video sequence, and the sample history audio also includes audio from a second time period adjacent to the sample audio block to be generated in the sample audio sequence.

[0257] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0258] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0259] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0260] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0261] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0262] like Figure 9As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0263] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0264] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as content generation methods and content generation model training methods. For example, in some embodiments, the content generation methods and content generation model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the content generation methods and content generation model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured in any other suitable manner (e.g., by means of firmware) to perform a content generation method and a training method for a content generation model.

[0265] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0266] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0267] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0268] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0269] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0270] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0271] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0272] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A content generation method, comprising: Acquire text data, which describes the audio content and video content to be generated; Based on the text data, the following operations are performed block by block in a streaming manner to generate audio and video sequences: A self-attention operation is performed based on the query vector of video noise for the video block to be generated, the query vector of audio noise for the audio block to be generated, the extended key vector, and the extended value vector to update the video noise and the audio noise. The extended key vector includes the key vectors of the video noise, the audio noise, historical video, historical audio, and the text block. The extended value vector includes the value vectors of the video noise, the audio noise, the historical video, the historical audio, and the text block. The text block is part or all of the text data.

2. The method according to claim 1, wherein, The historical video includes video within a first time period at the starting position of the video sequence, and the historical audio includes audio within the first time period at the starting position of the audio sequence.

3. The method according to claim 2, wherein, The historical video also includes video from a second time period adjacent to the video block to be generated in the video sequence, preceding the video block to be generated. The historical audio also includes audio from a second time period adjacent to the audio block to be generated in the audio sequence, preceding the audio block to be generated.

4. The method according to any one of claims 1 to 3, wherein, The video block and the audio block to be generated are generated through single-step inference.

5. The method according to any one of claims 1 to 4, wherein, The video and audio blocks to be generated are generated using a target content generation model. This target content generation model is obtained by updating the parameters of an initial content generation model based on a loss. The loss is determined based on the differences between the sample audio and video sequences and the target audio and video sequences. The sample audio and video sequences are obtained by inputting sample text into the initial content generation model and performing the following operations block by block using the initial content generation model: A self-attention operation is performed based on the query vector, sample expansion key vector, and sample expansion value vector of the sample video noise and the sample audio noise of the sample audio block to be generated, to update the sample video noise and the sample audio noise. The sample expansion key vector includes the key vectors of the sample video noise, the sample audio noise, the sample historical video, the sample historical audio, and the sample text block. The sample expansion value vector includes the value vectors of the sample video noise, the sample audio noise, the sample historical video, the sample historical audio, and the sample text block. The sample text block is part or all of the sample text, and the sample text describes the sample audio content and the sample video content to be generated.

6. A method for training a content generation model, comprising: Obtain sample text, which describes the audio and video content to be generated; The sample text is input into the initial content generation model, and the initial content generation model is used to perform the following operations block by block to obtain the sample audio sequence and sample video sequence: A self-attention operation is performed based on the query vector of the sample video noise of the sample video block to be generated and the query vector of the sample audio noise of the sample audio block to be generated, the sample expansion key vector, and the sample expansion value vector to update the sample video noise and the sample audio noise. The sample expansion key vector includes the key vectors of the sample video noise, the sample audio noise, the sample historical video, the sample historical audio, and the sample text block. The sample expansion value vector includes the value vectors of the sample video noise, the sample audio noise, the sample historical video, the sample historical audio, and the sample text block. The sample text block is part or all of the sample text. The loss is determined based on the differences between the sample audio sequence, the sample video sequence, the target audio sequence, and the target video sequence. The parameters of the initial content generation model are updated based on the loss to obtain the target content generation model.

7. The method according to claim 6, wherein, The differences between the sample audio sequences, sample video sequences, and target audio sequences and target video sequences are determined based on the divergence between the distributions of the sample audio sequences and sample video sequences and the distributions of the target audio sequences and target video sequences, and the updating of the parameters of the initial content generation model based on the loss includes: The parameters of the initial content generation model are updated based on the loss using model distillation. The initial content generation model is a student model, and the target audio sequence and the target video sequence are obtained by inputting the sample text into the teacher model.

8. The method according to claim 7, wherein, The teacher model is a multi-step denoising model, and the student model is a single-step inference model.

9. The method according to any one of claims 6 to 8, wherein, The sample text was generated using a large model.

10. The method according to claim 9, wherein, The sample text was generated by the large model based on the prompt word template of the target scene category.

11. The method according to any one of claims 6 to 10, wherein, The sample historical video includes the video within a first time period at the starting position of the sample video sequence, and the sample historical audio includes the audio within the first time period at the starting position of the sample audio sequence.

12. The method according to claim 11, wherein, The sample history video also includes video from a second time period adjacent to the sample video block to be generated in the sample video sequence, and the sample history audio also includes audio from a second time period adjacent to the sample audio block to be generated in the sample audio sequence.

13. A content generation apparatus, comprising: The acquisition module is used to acquire text data, which describes the audio content and video content to be generated; The generation module is used to generate audio and video sequences by performing the following operations block by block in a streaming manner based on the text data: A self-attention operation is performed based on the query vector of video noise for the video block to be generated, the query vector of audio noise for the audio block to be generated, the extended key vector, and the extended value vector to update the video noise and the audio noise. The extended key vector includes the key vectors of the video noise, the audio noise, historical video, historical audio, and the text block. The extended value vector includes the value vectors of the video noise, the audio noise, the historical video, the historical audio, and the text block. The text block is part or all of the text data.

14. A training device for a content generation model, comprising: The acquisition module is used to acquire sample text, which describes the audio content and video content to be generated. The generation module is used to input the sample text into the initial content generation model, and then use the initial content generation model to perform the following operations block by block to obtain the sample audio sequence and the sample video sequence: A self-attention operation is performed based on the query vector of the sample video noise of the sample video block to be generated and the query vector of the sample audio noise of the sample audio block to be generated, the sample expansion key vector, and the sample expansion value vector to update the sample video noise and the sample audio noise. The sample expansion key vector includes the key vectors of the sample video noise, the sample audio noise, the sample historical video, the sample historical audio, and the sample text block. The sample expansion value vector includes the value vectors of the sample video noise, the sample audio noise, the sample historical video, the sample historical audio, and the sample text block. The sample text block is part or all of the sample text. The loss determination module is used to determine the loss based on the differences between the sample audio sequence, the sample video sequence, the target audio sequence, and the target video sequence. An update module is used to update the parameters of the initial content generation model based on the loss, so as to obtain the target content generation model.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1 to 5 or claims 6 to 12.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 5 or claims 6 to 12.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5 or claims 6 to 12.

Images