Visual content generation method and apparatus, and computing device and storage medium

Abstract

Provided in the present application are a visual content generation method and apparatus, and a computing device and a storage medium. The method comprises: executing the iteration of N time steps on a diffusion model, so as to generate first visual content, wherein the N time steps comprise a key time step and a non-key time step, the key time step indicates the execution of complete network computation of the diffusion model, the non-key time step indicates the execution of partial network computation of the diffusion model, and an intermediate feature generated from the complete network computation corresponding to the key time step is used as an input of the partial network computation corresponding to the non-key time step. In the method, the intermediate feature generated at the key time step is reused at the non-key time step, and thus the amount of computation can be reduced and the generation speed of visual content can be increased.

Description

Visual content generation methods, apparatus, computing devices and storage media

[0001] This application claims priority to Chinese Patent Application No. 202410920671.3, filed on July 10, 2024, entitled "Visual Content Generation Method, Apparatus, Computing Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method, apparatus, computing device and storage medium for generating visual content. Background Technology

[0003] The Diffusion Model (DM) is a generative model used to intelligently generate visual content such as images or videos. Taking text-to-image generation as an example, during the inference phase, the diffusion model progressively denoises a purely noisy image (such as a Gaussian noise map) to ultimately generate an image that matches the text description input by the user.

[0004] However, the aforementioned stepwise denoising process requires multiple iterative calculations (typically 50-100 steps) on the entire diffusion model. Each iteration removes only a portion of the noise, and these multi-step iterations are executed sequentially. Therefore, the aforementioned visual content generation method based on the diffusion model requires a large amount of computation and is time-consuming, making it difficult to meet user needs in terms of visual content generation speed. Summary of the Invention

[0005] This application provides a method, apparatus, computing device, and storage medium for generating visual content, which can improve the speed of visual content generation.

[0006] Firstly, this application provides a visual content generation method, which includes: performing N time-step iterations on a diffusion model to generate first visual content. The N time steps include key time steps and non-key time steps. Key time steps indicate the execution of the complete network computation of the diffusion model, while non-key time steps indicate the execution of partial network computations of the diffusion model. Intermediate features generated by the complete network computation corresponding to the key time steps are used as input to the partial network computation corresponding to the non-key time steps.

[0007] In this scheme, by setting key time steps and non-key time steps in the N time steps (N is a positive integer greater than 1) that the diffusion model needs to iterate, and reusing the intermediate features generated by the complete network computation corresponding to the key time step in the non-key time step, it is possible to reduce the amount of computation and improve the speed of visual content generation (shorten the visual content generation latency) by not performing the complete network computation of the diffusion model in the non-key time step. This improves the user experience.

[0008] Based on the first aspect, in a possible implementation, before performing the aforementioned N time-step iterations on the diffusion model, the method further includes: performing M time-step iterations on the diffusion model to generate second visual content; if the error between the intermediate features generated by the complete network computation corresponding to the first time step and the intermediate features generated by the complete network computation corresponding to the second time step is less than or equal to an error threshold, then the time step corresponding to the first time step in the N time steps is determined as a critical time step, and the time step corresponding to the second time step in the N time steps is determined as a non-critical time step. Each of the M time steps performs a complete network computation of the diffusion model, and the M time steps include a first time step and a second time step, with the first time step preceding the second time step.

[0009] In this scheme, the diffusion model is iterated for M time steps in advance (each of the M time steps performs a complete network computation), and the error between the intermediate features generated by different time steps is calculated. This determines the similarity between the intermediate features generated by different time steps (error less than or equal to an error threshold). Then, the time steps corresponding to the N time steps with similarity are designated as key time steps and non-key time steps, respectively. The time steps executed earlier in the order are designated as key time steps, and the time steps executed later in the order are designated as non-key time steps. This allows the intermediate features generated by key time steps to be directly reused in non-key time steps when iterating the same diffusion model for the N time steps in subsequent iterations, thereby reducing the computational load of non-key time steps and thus reducing the overall computational load of the diffusion model iteration (lowering computational resource requirements) and improving the speed of visual content generation.

[0010] Based on the first aspect, in a possible implementation, before performing the aforementioned N time-step iterations on the diffusion model, the method further includes: performing M time-step iterations on the diffusion model to generate second visual content; determining M errors based on the intermediate features generated by the complete network computation corresponding to each of the M time steps, and determining the time steps corresponding to K of the M errors in the N time steps as key time steps. Wherein, K is less than M, each of the M time steps performs a complete network computation of the diffusion model, the M errors correspond one-to-one with the M time steps, and the error corresponding to each of the M time steps is the error between the intermediate features generated by the complete network computation corresponding to each time step and the intermediate features generated by the complete network computation corresponding to the next time step.

[0011] In this scheme, the diffusion model is iterated for M time steps beforehand (each of the M time steps involves a complete network computation), and the error between the intermediate features generated by each of these M time steps and its next time step is calculated. Then, based on the calculated multiple errors (each error corresponds to one of the M time steps), the time steps corresponding to some of the errors in the above N time steps are determined as critical time steps, and the next time step of the critical time step is set as a non-critical time step. This allows the intermediate features generated by the critical time steps to be directly reused in the non-critical time steps when the same diffusion model is iterated for the above N time steps in a subsequent run, thereby reducing the computational load of the non-critical time steps, thus reducing the overall computational load of the diffusion model iteration, and improving the speed of visual content generation.

[0012] Based on the first aspect, in a possible implementation, the aforementioned K errors are the K smallest among multiple errors. That is, the calculated multiple errors can be arranged in descending or ascending order, and then the K smallest errors can be selected. The time steps corresponding to these K errors within the aforementioned N time steps are then determined as critical time steps. K represents the number of critical time steps; the larger K is, the more critical time steps are set. The value of K can be set and adjusted according to usage requirements (such as accuracy requirements), and this application does not limit this.

[0013] Based on the first aspect, in a possible implementation, the value of K is specified by the user. That is, the user can specify the number of key time steps, and then the system automatically determines the corresponding number of key time steps from N time steps based on the user-specified value of K.

[0014] Based on the first aspect, in a possible implementation, the number of non-critical time steps among the aforementioned N time steps is multiple. The intermediate features generated by the complete network computation corresponding to the critical time step are used as inputs to the partial network computations corresponding to these multiple non-critical time steps, and the partial network computations corresponding to these multiple non-critical time steps are executed in parallel. That is, the intermediate features generated by the critical time step can be reused in multiple non-critical time steps, and the partial network computations corresponding to these multiple non-critical time steps can be executed in parallel, so as to further shorten the visual content generation latency and improve the visual content generation speed.

[0015] Based on the first aspect, in a possible implementation, when the iteration of the aforementioned N time steps reaches a critical time step, the complete network computation of the diffusion model is performed, and the intermediate features generated by the complete network computation corresponding to the critical time step are written into the first storage area. When the iteration of the N time steps reaches a non-critical time step, the intermediate features generated by the complete network computation corresponding to the critical time step are obtained from the first storage area, and partial network computation of the diffusion model is performed based on the obtained intermediate features generated by the complete network computation corresponding to the critical time step.

[0016] In other words, the intermediate features generated at key time steps are stored in the first storage area (such as a cache, which is not limited in this application). Then, when a non-key time step is reached, the intermediate features generated at the key time step can be obtained directly from the first storage area to perform the network computation corresponding to the non-key time step, thereby reducing the amount of computation and speeding up the generation of visual content.

[0017] Based on the first aspect, in a possible implementation, the output of the network computation corresponding to the non-critical time step includes the prediction noise of the non-critical time step. When the non-critical time step is reached, the first intermediate visual content is first denoised based on the noise data to obtain the denoised first intermediate visual content. Then, the denoised first intermediate visual content is denoised based on the prediction noise of the non-critical time step and the diffusion model to generate the intermediate visual content output by the non-critical time step.

[0018] It should be understood that since non-critical time steps reuse intermediate features generated by critical time steps for some network computations, and do not directly compute intermediate features, there may be some errors. Here, the first intermediate visual content output by the critical time step is first denoised, and then the denoised first intermediate visual content is denoised based on the predicted noise output by the non-critical time steps, thereby improving the texture details of the subsequently generated visual content.

[0019] Based on the first aspect, in a possible implementation, the intermediate features generated by the complete network computation corresponding to the critical time step are the outputs of the first downsampling module and the second upsampling module, and the partial network computation corresponding to the non-critical time step includes the computation of the first upsampling module. Specifically, the first downsampling module is the one with the largest output feature size among the multiple downsampling modules in the diffusion model, the first upsampling module is the one with the largest output feature size among the multiple upsampling modules in the diffusion model, and the second upsampling module is the one with the second largest output feature size among the multiple upsampling modules.

[0020] In this scheme, the intermediate features generated by the first downsampling module and the second upsampling module in the diffusion model at critical time steps are reused in non-critical time steps as input to the first upsampling module at non-critical time steps, thereby calculating the prediction noise at non-critical time steps. Therefore, only the first upsampling module needs to be loaded and its calculation needs to be executed at non-critical time steps. All other upsampling and downsampling modules in the diffusion model, except for the first upsampling module, do not need to be calculated at non-critical time steps (no need to load these modules), thereby reducing the computational load at non-critical time steps and shortening the visual content generation latency.

[0021] Based on the first aspect, in possible implementations, the first visual content is an image or video.

[0022] Secondly, this application also provides a visual content generation apparatus, including a functional module for performing a visual content generation method as described in the first aspect or any possible implementation thereof.

[0023] Thirdly, this application also provides a computing device including a processor and a memory. The processor is configured to execute instructions stored in the memory to cause the computing device to perform a visual content generation method as described in the first aspect or any possible implementation thereof.

[0024] Fourthly, this application also provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs a visual content generation method as described in the first aspect or any possible implementation thereof.

[0025] Fifthly, this application also provides a chip system, the chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute a visual content generation method as described in the first aspect or any possible implementation thereof.

[0026] In a sixth aspect, this application also provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices (including at least one computing device), perform a visual content generation method as described in the first aspect or any possible implementation thereof.

[0027] In a seventh aspect, this application also provides a computer program product containing instructions. When the instructions are executed by a cluster of computing devices (including at least one computing device), the cluster of computing devices causes the cluster of computing devices to perform a visual content generation method as described in the first aspect or any possible implementation thereof.

[0028] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments are briefly introduced below.

[0030] Figure 1 is a schematic diagram of the structure of a U-Net network provided in an embodiment of this application;

[0031] Figure 2 is a schematic diagram of an image generation process based on a diffusion model provided in an embodiment of this application;

[0032] Figure 3 is a schematic diagram of calculating the error between intermediate features extracted at different time steps according to an embodiment of this application;

[0033] Figure 4 is a schematic diagram of another method for calculating the error between intermediate features extracted at different time steps, provided by an embodiment of this application.

[0034] Figure 5 is a flowchart illustrating a visual content generation method provided in an embodiment of this application;

[0035] Figure 6 is a schematic diagram of using intermediate features extracted at one time step in the next time step according to an embodiment of this application;

[0036] Figure 7 is a schematic diagram of using intermediate features extracted at one time step in multiple subsequent time steps according to an embodiment of this application;

[0037] Figure 8 is a schematic diagram of parallel computing performed at multiple time steps according to an embodiment of this application;

[0038] Figure 9 is a schematic diagram of the structure of a visual content generation device provided in an embodiment of this application;

[0039] Figure 10 is a schematic diagram of the structure of a computing device provided in this application;

[0040] Figure 11 is a schematic diagram of a computing device cluster provided in this application;

[0041] Figure 12 is a schematic diagram of the two computing devices provided in this application connected via a network. Detailed Implementation

[0042] To facilitate understanding of the technical solution of this application, some of the terms and concepts involved in this application will be briefly introduced first.

[0043] 1. Diffusion Model (DM): A generative model based on the diffusion process, used to generate visual content such as images or videos. Taking image generation as an example, the training process of the diffusion model includes forward diffusion and backward diffusion. First, noise is gradually added to the image during forward diffusion. Then, in the backward diffusion process (also known as denoising or reverse generation), the model learns how to gradually remove the noise added to the image. Once the diffusion model is trained, it can start with a purely noisy image and iteratively denoise it step-by-step through the diffusion model's calculations. Each step is based on the denoising output of the previous step, ultimately generating an image that meets the conditional information (used to guide the generation of what kind of image).

[0044] 2. Stable diffusion (SD) model: A common diffusion model, including variational autoencoder (VAE), noise prediction network, denoiser, and conditional encoder.

[0045] The VAE comprises an encoder and a decoder. The encoder compresses the original visual content in pixel space into a low-dimensional latent space, resulting in a latent representation of the original visual content. This latent representation contains compressed information about the original visual content but is much smaller in size, essentially a compressed version of the original visual content. To distinguish it from the pixel space visual content, this application refers to the latent representation of the pixel space visual content as latent space visual content (or intermediate visual content). The decoder restores the latent space visual content back to the pixel space visual content.

[0046] The noise prediction network is used to predict noise. Its training process includes a forward diffusion process and a backward diffusion process in the latent space: first, noise is gradually added to the visual content in the latent space during the forward diffusion process, and then during the backward diffusion process, the noise prediction network learns how to gradually predict the noise added to the visual content in the latent space.

[0047] A denoiser (also called a scheduler) is used to progressively denoise latent space visual content based on the noise predicted by the noise prediction network.

[0048] The conditional encoder is used to convert conditional information (also known as cue information) into corresponding text representations to guide the visual content generation process of the diffusion model, ensuring that the visual content generated by the diffusion model matches the conditional information. This conditional information can be text, visual content, segmentation maps, or other types of information; this application does not limit its scope.

[0049] For example, a contrastive language-image pre-training (CLIP) model can be used as a conditional encoder. CLIP is a model pre-trained on large-scale visual content-text pair data, which includes a text encoder and a visual content encoder. The text encoder is used to convert text into a numerical representation, i.e., a text representation vector or text embedding, while the visual content encoder is used to convert visual content into a corresponding visual content representation vector.

[0050] 3. U-Net: A deep neural network named for its network architecture resembling the uppercase letter U. It is commonly used as a noise prediction network in SD models. A brief introduction to the structure of U-Net follows.

[0051] Please refer to Figure 1. Figure 1 is a schematic diagram of the structure of a U-Net network provided in an embodiment of this application. The U-Net network has an encoder-decoder structure, and the decoder and encoder are connected by a connector.

[0052] The encoder part of U-Net includes multiple down-sample blocks, each used to extract low-level features from the input data. Different down-sample blocks extract different features from the input data, thus capturing multi-scale feature information. The down-sample blocks are sequentially connected, with the feature sizes extracted by each block decreasing progressively according to this sequential connection. Assuming U-Net includes K1 down-sample blocks, Di is the i-th down-sample block in U-Net (i∈[1,K1]), D represents an up-sample block, and i represents the i-th feature size extracted by Di among all down-sample blocks (arranged in descending order). The features extracted by one down-sample block serve as the input to the next down-sample block, which further extracts features based on the features extracted by the previous down-sample block. Each down-sample block may contain convolutional layers, activation functions, pooling layers, etc. This application does not specifically limit the internal structure of the down-sample blocks.

[0053] The decoder part of U-Net includes multiple up-sample blocks, each used to extract high-level features from the input data. These up-sample blocks are sequentially connected, with the feature sizes extracted by each block increasing sequentially. Assuming U-Net contains K² up-sample blocks, and Uj is the j-th up-sample block (j∈[1,K²]), where U represents the up-sample block and j represents the j-th feature size extracted by Uj (arranged in descending order). The features extracted by one up-sample block are used as input to the next up-sample block. There is a one-to-one correspondence between the up-sample blocks and the down-sample blocks; each up-sample block is connected to its corresponding down-sample block (a connection known in the industry as a skip connection), allowing the input of features from the corresponding down-sample block to the up-sample block, thus fusing low-level and high-level features. Each up-sample block may contain a deconvolutional layer (also called a transposed convolution), an activation function, and an up-sampled layer. This application does not impose specific limitations on the number of upsampling modules or their internal structure; these can be set and adjusted according to actual usage requirements.

[0054] It should be noted that Figure 1 only uses the three downsampling modules D1 to D3 and the three downsampling modules U1 to U3 as examples, and does not constitute a specific limitation. In actual application scenarios, the number, internal structure, connection relationship, etc. of the upsampling and downsampling modules can be adjusted according to the usage requirements.

[0055] The following is a brief introduction to the application scenarios involved in this application.

[0056] Please refer to Figure 2, which is a schematic diagram of an image generation process based on a diffusion model according to an embodiment of this application. First, a Gaussian noise map in the latent space is constructed based on a random seed. Then, starting from this Gaussian noise map (as the model input), the diffusion model iteratively calculates to progressively denoise the Gaussian noise map, ultimately generating an image that satisfies the input condition information. The condition information can be text, image, segmentation map, etc., and this application does not limit it.

[0057] Specifically, taking text as the conditional information as an example, suppose the input text is "a standing person". This text is input into the conditional encoder, which encodes the text to obtain the corresponding representation vector, which is used to guide the subsequent image generation process. The Gaussian noise map of the latent space is used as the initial latent space image (denoted as x0), and the following steps S101 to S102 are repeated until the current time step t (assuming the initial value of t is 0) reaches the number of iterations T set for this iteration process.

[0058] S101: At time step t, the representation vector and the current latent space image x are... t The current time step t and its identifier (such as the time step number) are input into the noise prediction network in the diffusion model. The noise prediction network outputs the predicted noise at time step t. Taking the U-Net network in Figure 1 as an example, D1~D3 and U3~U2 in the U-Net network sequentially extract the input latent space image x. t The features extracted by D1 and U2 (referred to as intermediate features in this application) are then used to calculate the predicted noise at time step t.

[0059] S102: Based on the predicted noise at time step t, use the denoiser in the diffusion model to adjust x. t Denoising is performed to generate the latent space image x at time step t. t+1 (The latent space variable is used as input for the next time step). Then update the value of t (t = t + 1).

[0060] Finally, using the VAE decoder in the diffusion model, the latent space image (intermediate product, equivalent to a compressed version of the pixel space image) obtained in the last iteration is restored to a pixel space image that satisfies the above-mentioned input text.

[0061] It should be noted that in the above stepwise denoising process, the entire diffusion model requires multiple time-step iterations, and these iterations are executed sequentially. The model can only begin calculating at the next time step after the previous time-step's calculation is complete. This results in a slow speed for the diffusion model in generating visual content such as images and videos, making it difficult to meet user needs and limiting the application scope of the diffusion model.

[0062] To address the slow speed of visual content generation in diffusion modules, this application provides a visual content generation method. This method sets key and non-key time steps within the N time steps required for the diffusion model to generate the first visual content. Key time steps indicate the execution of the complete network computation of the diffusion model, while non-key time steps indicate the execution of partial network computation. The intermediate features generated by the complete network computation at the key time step are used as input for the partial network computation at the non-key time step. This eliminates the need for the complete network computation of the diffusion model at non-key time steps, requiring only partial computation, thereby reducing computational load and improving the speed of visual content generation. The visual content generation method is described in detail below.

[0063] The following section will explain how to determine the critical and non-critical time steps among the above N time steps.

[0064] First, a diffusion model is selected. This diffusion model can be user-selected or a default one, used to perform a certain visual content generation task. This visual content generation task can be one of the following: text-to-image, image-to-image, text-to-video, or image-to-video tasks; this application does not limit this. Specifically, a text-to-image task generates an image from text, where the conditional information type is text, and the generated visual content type is an image. An image-to-image task generates another image from an image, such as generating a cartoon image from a photograph; where the conditional information type is image, and the generated visual content type is also image. A text-to-video task generates a video from text, where the conditional information type is text, and the generated visual content type is video. An image-to-video task generates a video from an image, where the conditional information type is image, and the generated visual content type is video. For ease of description, the following text will primarily use the text-to-image task as an example.

[0065] Then, the diffusion model is iterated for M time steps (M is a positive integer greater than 1) to generate the second visual content. Each of these M time steps performs the complete network computation of the diffusion model, that is, each time step completes the full computation of the diffusion model without omitting any computational processes.

[0066] Next, based on the intermediate features generated by the complete network computation corresponding to each of these M time steps, the critical and non-critical time steps among the above N time steps are determined, including the following method one and method two.

[0067] Method 1: If the error between the intermediate features generated by the complete network computation corresponding to the first time step and the intermediate features generated by the complete network computation corresponding to the second time step is less than or equal to an error threshold, then the time step corresponding to the first time step among N time steps is determined as a critical time step, and the time step corresponding to the second time step among N time steps is determined as a non-critical time step. Here, the M time steps include the first time step and the second time step, with the first time step preceding the second time step. M may be greater than, equal to, or less than N; this application does not specifically limit this. M and / or N can be specified by the user or can be default values; this application also does not limit this.

[0068] The following section provides a detailed explanation of Method 1, starting with how to determine the error between intermediate features generated at different time steps.

[0069] As shown in Figure 1, the U-Net network can be used as a noise prediction network in the diffusion model. Each downsampling module and each upsampling module in the U-Net network can extract intermediate features. Specifically, the error is calculated based on the intermediate features extracted by which of the sampling modules (both upsampling and downsampling modules can be referred to as sampling modules) at different time steps. This depends on which of the sampling modules in the U-Net network's computation process is to be skipped and which is to be retained. For ease of description, this application refers to one or more sampling modules whose computation process is to be retained as target sampling modules. The computation process of other sampling modules besides the target sampling modules can be appropriately omitted. That is, only a portion of the U-Net network computation (i.e., the computation of the target sampling modules) is performed, omitting another portion of the U-Net network computation (i.e., the computation of other sampling modules besides the target sampling modules).

[0070] Since the first upsampling module U1 in U-Net is used to output prediction noise, the calculation process of U1 cannot be omitted. Therefore, the target sampling module must include at least U1. Among them, U1 is the upsampling module in U-Net that generates the largest feature size. Of course, the target sampling module can also include other sampling modules, which can be selected according to the requirements. After selecting the target sampling module, the error between the intermediate features generated at different time steps can be calculated using the following formula (1): E(i,j)=∑ q f_error(F q_i F q_j ) Formula (1)

[0071] Here, time step i and time step j are two different time steps among the M time steps mentioned above, and the iteration calculation order of time step i is earlier than that of time step j. q∈[1,Q], where Q is the number of associated sampling modules of the target sampling module. The input of the target sampling module includes the intermediate features generated by the associated sampling modules of the target sampling module, that is, the output of the associated sampling modules of the target sampling module is connected to the input of the target sampling module. F q_i F is the intermediate feature generated by the q-th associated sampling module of the target sampling module at time step i. q_i It is the intermediate feature generated by the q-th associated sampling module of the target sampling module at time step j, where f_error represents the error function, f_error(F n_i F n_j ) represents the use of the error function to calculate F q_i and F q_i The error between them. ∑ q f_error(F q_i F q_j ) represents the sum of errors between the intermediate features generated by each of the Q associated sampling modules at time step i and the intermediate features generated at time step j.

[0072] Optionally, the above f_error can be one or more of the following error functions: mean square error (QSE), root mean square error (QSE), mean absolute error (QAE), etc. This application does not make any specific limitation on this.

[0073] It should be understood that there are multiple ways to select the target sampling module. The following explanation will use cases one through three as examples.

[0074] Case 1: Select the first upsampling module U1 in the U-Net network as the target sampling module.

[0075] As described in the previous introduction to Figure 1, the input of the first upsampling module U1 in U-Net includes the intermediate features generated by the first downsampling module D1 and the intermediate features generated by the second upsampling module U2. D1 is the downsampling module in U-Net that generates the largest feature size, and U2 is the upsampling module in U-Net that generates the second largest feature size. Therefore, when the target sampling module only includes U1, the associated sampling modules of the target sampling module are D1 and U2, and the number of associated sampling modules of the target sampling module is 2. At this time, formula (1) can be rewritten as formula (2) to calculate the error between the intermediate features generated at different time steps. E(i,j)=f_error(F D1_i FD1_j )+f_error(F U2_i F U2_j ) Formula (2)

[0076] Here, time step i and time step j are two different time steps among the above M time steps, and the iteration calculation order of time step i is earlier than that of time step j, F D1_i F represents the intermediate feature generated by D1 at time step i. D1_j F represents the intermediate feature generated by D1 at time step j. U2_i F represents the intermediate feature generated by U2 at time step i. U2_j For the intermediate features generated by U2 at time step j, f_error represents the error function, f_error(F D1_i F D1_j ) indicates that F is calculated based on the error function. D1_i and F D1_j The error between them, f_error(F U2_i F U2_j ) indicates that F is calculated based on the error function. U1_i and F U1_j The error between them.

[0077] Taking U-Net in Figure 1 as an example, assuming M = 50, i = 40, j = 41, for time step 40 and time step 41 out of 50 time steps, the error between the intermediate features they produce is E(40,41) = f_error(F D1_40 F D1_41 )+f_error(F U2_40 F U2_41 ).

[0078] Scenario 2: Select the first upsampling module U1 and the second upsampling module U2 in the U-Net network as the target sampling modules.

[0079] As shown in Figure 1, if U1 and U2 are considered as a whole (as the target sampling module), the input of this target sampling module includes the intermediate features generated by the first downsampling module D1, the intermediate features generated by the second downsampling module D2, and the intermediate features generated by the third upsampling module U3. Among them, D1 is the downsampling module that generates the largest feature size in U-Net, D2 is the downsampling module that generates the second largest feature size in U-Net, and U3 is the upsampling module that generates the third largest feature size in U-Net. Therefore, when the target sampling module only includes U1 and U2, the associated sampling modules of the target sampling module are D1, D2, and U3, and the number of associated sampling modules of the target sampling module is 3. At this time, formula (1) can be rewritten as formula (3) to be used for the error between intermediate features generated at different time steps. E(i,j)=f_error(F D1_i F D1_j )+f_error(F D2_i F D2_j )+f_error(F U3_i F U3_j ) Formula (3)

[0080] Here, time step i and time step j are two different time steps among the above M time steps, and the iteration calculation order of time step i is earlier than that of time step j, F D1_i F represents the intermediate feature generated by D1 at time step i. D1_j F represents the intermediate feature generated by D1 at time step j. D2_i F represents the intermediate feature generated by D2 at time step i. D2_j F represents the intermediate feature generated by D2 at time step j. U3_i F represents the intermediate feature generated by U3 at time step i. U3_j For the intermediate features generated by U3 at time step j, f_error represents the error function, f_error(F D1_i F D1_j ) indicates that F is calculated based on the error function. D1_i and F D1_j The error between them, f_error(F D2_i F D2_j ) indicates that F is calculated based on the error function. D2_i and F D2_j The error between them, f_error(F U3_i F U3_j ) indicates that F is calculated based on the error function. U3_i and F U3_j The error between them.

[0081] Case 3: The target sampling module includes the first downsampling module D1 and the first upsampling module U1.

[0082] As shown in Figure 1, if D1 and U1 are considered as a whole (as a target sampling module), at time step t, the input of this target sampling module includes the current visual content x. t (This is the visual content output at the previous time step t-1) and the intermediate features generated by the second upsampling module U2. Therefore, when the target sampling module only includes D1 and U1, the associated sampling module of the target sampling module is only U2, and the number of associated sampling modules of the target sampling module is 1. At this time, formula (1) can be rewritten as formula (4) to account for the error between intermediate features generated at different time steps. E(i,j)=f_error(F U2_i F U2_j ) Formula (4)

[0083] Here, time step i and time step j are two different time steps among the above M time steps, and the iteration calculation order of time step i is earlier than that of time step j, F U2_i F represents the intermediate feature generated by U2 at time step i. U2_j For the intermediate features generated by U2 at time step j, f_error represents the error function, f_error(F U2_i F U2_j ) indicates that F is calculated based on the error function. U2_i and F U2_j The error between them.

[0084] It should be noted that scenarios one through three above are merely examples and do not constitute specific limitations. In practical applications, the target sampling module can have other options. It should be understood that the fewer sampling modules in the target sampling module, the more sampling modules are skipped in U-Net's computation, resulting in better acceleration of visual content generation. The appropriate target sampling module can be selected based on usage requirements.

[0085] The previous section explained how to determine the errors between intermediate features generated at different time steps. The following section introduces how Method 1 determines the critical and non-critical time steps among the above N time steps based on these errors.

[0086] For time step i and time step j in the above M time steps, if the error between the intermediate feature generated by time step i and the intermediate feature generated by time step j is less than or equal to the error threshold, then the time step corresponding to time step i in the above N time steps is determined as a critical time step, and the time step corresponding to time step j in the above N time steps is determined as a non-critical time step. Here, time step i is earlier than time step j, and time step i and time step j may be adjacent (separated by 0 time steps) or not adjacent (separated by one or more time steps); this application does not impose any limitation on this.

[0087] The aforementioned error thresholds can be set by the user, can be default values, or can be optimized and adjusted; this application does not limit this. For example, by evaluating the quality (such as sharpness, accuracy, etc.) of the visual content generated by the diffusion model under multiple different error thresholds, the error threshold corresponding to higher quality is determined as the optimal error threshold, which is then used to determine the critical and non-critical time steps among the aforementioned N time steps.

[0088] It should be noted that for time step i in the aforementioned M time steps, assuming that the corresponding time step i in the aforementioned N time steps is time step i', and that time step i is ranked i-th in the execution order (arranged from earliest to latest) of the M time steps, then time step i' should also be ranked i-th in the execution order (arranged from earliest to latest) of the N time steps. Similarly, for time step j in the aforementioned M time steps, assuming that the corresponding time step j in the aforementioned N time steps is time step j', and that time step j is ranked j-th in the execution order (arranged from earliest to latest) of the M time steps, then time step j' should also be ranked j-th in the execution order (arranged from earliest to latest) of the N time steps. In other words, the correspondence between the time steps in the aforementioned M time steps and the time steps in the aforementioned N time steps refers to the correspondence in execution order; that is, time steps in the same execution order have a corresponding relationship.

[0089] For example, suppose N=60 and M=50, that is, the diffusion model performs 60 time steps when generating the first visual content and 50 time steps when generating the second visual content.

[0090] For ease of description, the iteration of the above 60 time steps constitutes the first iteration process. These 60 time steps are arranged in execution order from earliest to latest (the normal order before critical and non-critical time steps are determined). Here, time step n_i represents the i-th time step among these 60 time steps in the first iteration process. Similarly, the iteration of the above 50 time steps constitutes the second iteration process. These 50 time steps are arranged in execution order from earliest to latest, and time step m_i represents the i-th time step among these 50 time steps in the second iteration process.

[0091] Since the order of time step n_i in the above 60 time steps is the same as the order of time step m_i in the above 50 time steps, there is a correspondence between time step n_i and time step m_i. Where i∈[1,50].

[0092] According to the requirements of Method 1, if the error between time step m_40 and time step m_41 (where i = 40 and j = 41) in the above 50 time steps is less than or equal to the error threshold, then time step n_40 corresponding to time step m_40 in the above 60 time steps can be determined as a critical time step, and time step n_41 corresponding to time step m_41 in the above 60 time steps can be determined as a non-critical time step.

[0093] It should be understood that one or more critical time steps can be determined from the aforementioned N time steps using Method 1, and one or more non-critical time steps can also be determined from the aforementioned N time steps; this application does not specifically limit this. For example, the user can specify the number of critical time steps to be found, thereby automatically determining the corresponding number of critical time steps from the N time steps using Method 1.

[0094] Method 2: Based on the intermediate features generated by the complete network computation corresponding to each of the M time steps, M errors are determined. Each of the M errors corresponds one-to-one with one of the M time steps, and the error corresponding to each time step is the error between the intermediate feature generated by the complete network computation at that time step and the intermediate feature generated by the complete network computation at the next time step. Then, the time steps corresponding to K of the above M errors are determined as key time steps within N time steps, where K is less than M. The value of K can be specified by the user, be a default value, or be automatically determined; this application does not limit this.

[0095] For information on how to determine the error between intermediate features generated at different time steps in Method 2, please refer to the relevant introduction in Method 1, which will not be repeated here.

[0096] Optionally, the M errors are sorted in ascending order, and the time steps corresponding to the top K errors are selected. These K selected time steps (located among the M time steps) are then identified as critical time steps within the N time steps. Alternatively, the next time step after each critical time step can be designated as a non-critical time step. The intermediate features generated by each critical time step are used as input to the next non-critical time step to perform partial network computation.

[0097] Optionally, the critical time steps and non-critical time steps among the above N time steps can be determined through the following steps S201 to S204.

[0098] S201: Calculate the error between the intermediate feature generated at each of the above M time steps and the intermediate feature generated at the next time step.

[0099] For details on error calculation methods, please refer to the previous introduction; they will not be repeated here.

[0100] S202: Determine the two time steps corresponding to the minimum error calculated in step S201, and determine the time steps corresponding to these two time steps in the above N time steps as critical time steps and non-critical time steps, respectively.

[0101] Among these, the execution order of the critical time steps is before that of the non-critical time steps.

[0102] For example, as shown in Figure 3, assuming the above M time steps are 50 steps (M=50), based on the error calculation method introduced above, the error between the intermediate feature generated at each time step and the intermediate feature generated at its next time step is calculated. For example, E(46,47) represents the error between the intermediate feature generated at time step m_46 and the intermediate feature generated at time step m_47 in these 50 time steps, and E(47,48) represents the error between the intermediate feature generated at time step m_47 and the intermediate feature generated at time step m_48 in this iteration process.

[0103] Assuming that E(47,48) is the smallest error among all calculated errors, and E(47,48) corresponds to time steps m_47 and m_48 out of the aforementioned 50 time steps, then the time steps corresponding to these two time steps in the aforementioned N time steps can be designated as critical time steps and non-critical time steps, respectively. Specifically, time step m_47 corresponds to time step n_47 in the aforementioned N time steps, and time step n_47 is determined as a critical time step; time step m_48 corresponds to time step n_48 in the aforementioned N time steps, and time step n_48 is determined as a non-critical time step. The intermediate features generated by the critical time step n_47 will be used as the input for its next non-critical time step n_48.

[0104] To distinguish it from the other time steps in the above M time steps, time step m_47 is marked with a pentagram in Figure 3, indicating that time step m_47 already has a corresponding critical time step in the above N time steps. Time step m_48 is also marked with a circle in Figure 3, indicating that time step m_48 already has a corresponding non-critical time step in the above N time steps.

[0105] It should be noted that the first time step in Figure 3 is numbered starting from 1, but in practical applications, it can also start from 0. Figure 3 shows the 50 time steps arranged in ascending order of their time step numbers, meaning the smaller the number, the earlier the time step executes. In practical applications, they can also be arranged in descending order of their time step numbers, meaning the larger the number, the earlier the time step executes. This application does not impose a specific limitation on this arrangement.

[0106] S203: For the other time steps among the M time steps besides the corresponding non-critical time steps, determine the error between them and the time step of their next time step (excluding time steps with corresponding critical and non-critical time steps) in the intermediate features generated. Then, determine the time steps corresponding to the minimum error determined at this time in the above N time steps as critical and non-critical time steps respectively.

[0107] Among these, the execution order of the critical time steps is before that of the non-critical time steps.

[0108] Continuing from the previous example, we have already determined that time step m_47 out of M time steps has a corresponding critical time step in N time steps, and that time step m_48 has a corresponding non-critical time step in N time steps. Now we need to determine the errors in the intermediate features generated by the other time steps besides time step m_48 and their next time step without corresponding critical or non-critical time steps. As shown in Figure 4, since most of the errors have already been calculated in step S202, we only need to additionally calculate the errors of time steps m_47 and m_49 (time step m_48 is skipped because it already has a corresponding non-critical time step), i.e., E(47,49). Then, we determine the two time steps corresponding to the minimum error at this point, and then determine the time steps corresponding to these two time steps in N time steps as critical and non-critical time steps, respectively.

[0109] If the two time steps corresponding to the minimum error at this time are time step m_45 and time step m_46, then the time step corresponding to time step m_45 in N time steps (i.e., time step n_45) is determined as the critical time step, and the time step corresponding to time step m_46 in N time steps (i.e., time step n_46) is determined as the non-critical time step.

[0110] If the two time steps corresponding to the minimum error at this point are time steps m_47 and m_49, then the time step corresponding to time step m_49 in the N time steps (i.e., time step n_49) is determined as a non-critical time step. Combining the previously determined critical time step n_47 and non-critical time step n_48, it can be seen that the intermediate features generated by the critical time step n_47 can be used as input to both the non-critical time step n_48 and the non-critical time step n_49. That is, the intermediate features generated by the critical time step n_47 can be reused in the network computation corresponding to the non-critical time step n_48 and the critical time step n_49.

[0111] S204: Repeat step S203 above until the termination condition is met.

[0112] Optionally, the termination condition may be that the number of times step S203 is repeated reaches a first threshold, the number of identified critical time steps reaches a second threshold, the number of non-critical time steps reaches a third threshold, the ratio of critical time steps to non-critical time steps in all time steps (number T) is greater than or equal to a fourth threshold, or there are no errors less than or equal to the error threshold among all currently calculated errors, etc. The above thresholds can be reasonably set according to actual usage requirements, and this application does not limit them.

[0113] Based on the key and non-key time steps determined in the previous N time steps, the following describes how to generate visual content.

[0114] Please refer to Figure 5, which is a flowchart of a visual content generation method provided in an embodiment of this application, including steps S501 to S502.

[0115] S501: Perform N time-step iterations on the diffusion model, where the N time steps include critical time steps and non-critical time steps. The critical time steps indicate the execution of the complete network computation of the diffusion model, and the non-critical time steps indicate the execution of a partial network computation of the diffusion model. The intermediate features generated by the complete network computation corresponding to the critical time step are used as the input to the partial network computation corresponding to the non-critical time step.

[0116] Regarding the types of diffusion models, please refer to the previous introduction; this application does not impose specific limitations.

[0117] The critical and non-critical time steps among the above N time steps can be determined according to method one or method two introduced above. For details, please refer to the previous introduction, which will not be repeated here.

[0118] Taking the diffusion model for performing text-to-image tasks as an example, assuming N=60 and M=50, that is, the first iteration of the diffusion model contains 60 time steps and the second iteration of the diffusion model contains 50 time steps.

[0119] First, the second iteration of the diffusion model is executed. The 60 time steps in the second iteration are executed sequentially, with each time step performing the complete computation of the diffusion model. The noise prediction network (U-Net) in the diffusion model extracts corresponding intermediate features at each step. The specific iteration process can be seen in Figure 2, and will not be elaborated here. The U-Net in the diffusion model includes multiple upsampling modules and multiple downsampling modules. In time step m_47 of the second iteration, each module in the U-Net extracts intermediate features from the intermediate visual content (latent space visual content) output from time step m_46. In time step m_48 of the second iteration, each module in the U-Net extracts intermediate features from the intermediate visual content output from time step m_47.

[0120] Assuming U1 in U-Net is taken as the target sampling module, as introduced above, the associated sampling modules of U1 are D1 and U2. Then, the error between the intermediate features generated at time step m_47 and the intermediate features generated at time step m_48 can be calculated according to formula (2). If the error between the intermediate features generated at time step m_47 and the intermediate features generated at time step m_48 is less than or equal to the error threshold, then according to the provisions of Method 1, the time step n_47 corresponding to the above N time steps can be determined as the critical time step, and the time step n_48 corresponding to the above N time steps can be determined as the non-critical time step. The intermediate features generated by the above critical time steps will be used as the input of the above non-critical time steps to perform the partial network calculation corresponding to the non-critical time steps. This partial network calculation includes the calculation process of U1 in U-Net, but does not include the calculation process of other sampling modules in U-Net except for U1 (that is, the calculation process of most sampling modules in U1 is skipped).

[0121] Similarly, more critical and non-critical time steps can be determined within N time steps, which will not be elaborated here.

[0122] Specifically, when the iteration of the aforementioned N time steps reaches a critical time step, the complete network computation of the diffusion model is performed, and the intermediate features generated by the complete network computation corresponding to the critical time step are written to the first storage area. Then, when the iteration of the aforementioned N time steps reaches a non-critical time step, the intermediate features generated by the complete network computation corresponding to the critical time step are retrieved from the first storage area, and partial network computation of the diffusion model is performed based on the retrieved intermediate features generated by the complete network computation corresponding to the critical time step. Regarding the first storage area, it can be a cache or other type of storage area, and this application does not specifically limit it. This application also does not limit the location and size of the first storage area, and it can be set according to the usage requirements.

[0123] Optionally, the diffusion model (the noise prediction network in it) includes multiple upsampling modules and multiple downsampling modules, with each downsampling module connected to its corresponding upsampling module. The first intermediate feature includes the intermediate features generated by the complete network calculations corresponding to the first downsampling module D1 and the second upsampling module U2. The first downsampling module D1 is the downsampling module with the largest feature size extracted from the multiple downsampling modules, and the second upsampling module U2 is the upsampling module with the second largest feature size extracted from the multiple upsampling modules.

[0124] After obtaining the first intermediate feature at the critical time step, the prediction noise for the non-critical time step can be calculated using the first upsampling module U1 based on this first intermediate feature. Here, the first upsampling module U1 is the one with the largest feature size extracted from the multiple upsampling modules. In other words, in this case, U1 is the target sampling module, and D1 and U2 are associated sampling modules of U1. Therefore, the intermediate features generated by D1 and U2 at the critical time step can be used as the intermediate features generated by D1 and U2 at the non-critical time step (this is simply reused, not actually calculated). These intermediate features are then input into U1 to calculate the prediction noise for the non-critical time step. For details on the target sampling module and its associated sampling modules, please refer to the previous description; they will not be repeated here.

[0125] Continuing from the previous example, after determining the critical and non-critical time steps among the N time steps to be executed in the first iteration of the diffusion model, the first iteration process begins. As shown in Figure 6, when the critical time step n_47 of the first iteration process is reached, the intermediate features extracted by D1 and U2 in U-Net at the critical time step n_47 are written into the first storage area (such as a cache) for use in the non-critical time step n_48.

[0126] Then, when the process reaches the non-critical time step n_48 in the first iteration, the intermediate features extracted by D1 and U2 at the critical time step n_47 are retrieved from the cache and used as the intermediate features extracted by D1 and U2 at the non-critical time step n_48. These intermediate features are then input into U1 for computation, and U1 outputs the prediction noise for the non-critical time step n_48. It can be seen that, based on the indication of critical and non-critical time steps, the intermediate features extracted by D1 and U2 at the critical time step n_47 are directly reused in the non-critical time step n_48. This means that the non-critical time step n_48 only needs to execute the computation process of U1 in U-Net. Other sampling modules in U-Net besides U1 do not need to be computed in the non-critical time step n_48, thus reducing the computational load and improving the speed of visual content generation.

[0127] Optionally, the first intermediate visual content is denoised based on noisy data to obtain denoised first intermediate visual content. Then, the denoised first intermediate visual content is denoised based on the prediction noise and diffusion model of non-critical time steps to generate intermediate visual content output at non-critical time steps. The output of the complete network computation at the critical time step includes the aforementioned first intermediate visual content, while the output of the partial network computation at the non-critical time step includes the prediction noise of the non-critical time step.

[0128] It should be understood that since non-critical time steps reuse intermediate features generated by critical time steps for some network computations, and do not directly compute intermediate features, there may be some errors. Here, the first intermediate visual content output by the critical time step is first denoised, and then the denoised first intermediate visual content is denoised based on the predicted noise output by the non-critical time steps, thereby improving the texture details of the subsequently generated first visual content.

[0129] Regarding the aforementioned noise data, it can be the initial noise data x0 input at the first time step (such as Gaussian noise data in the latent space), and then the visual content x at time step t can be processed according to the following formula (5). t Add noise: x t * = x t +f(x0)=x t +(a*x0-m) Formula (5)

[0130] Wherein, time step t can be a critical time step or a non-critical time step among the above N time steps, x t * is the x after adding noise. t f represents a linear function, and a and m can be set according to usage requirements; this application does not impose any restrictions on them. For example, assuming a = 0.001 and m = 0, then x t * = x t +0.001x0.

[0131] Optionally, there may be multiple non-critical time steps among the N time steps. The intermediate features generated by the complete network computation corresponding to the critical time step are used as inputs for the partial network computations corresponding to multiple non-critical time steps, and these partial network computations corresponding to multiple non-critical time steps are executed in parallel.

[0132] For example, as shown in Figure 7, the encoder-decoder structure represented by two trapezoids in the figure represents the U-Net network in the diffusion model. The encoder includes multiple downsampling modules, and the decoder includes multiple upsampling modules. Here, it is assumed that the first upsampling module U1 in U-Net is taken as the target sampling module, and the associated sampling modules of U1 are the first downsampling module D1 and the second upsampling module U2.

[0133] Assume that time step t is the critical time step among N time steps, time steps t+1 and t+2 are non-critical time steps, and time step t+3 is the critical time step. That is, the previous critical time step for time steps t+1 and t+2 is time step t. When execution reaches time step t, since time step t is a critical time step, the complete network computation of U-Net will be performed at time step t, and the intermediate features generated by D1 and U2 at time step t will be written to the cache.

[0134] Then, when execution reaches time step t+1, since time step t+1 is a non-critical time step, the intermediate features generated by D1 and U2 at the previous critical time step (i.e., time step t) can be directly retrieved from the cache and input into U1 for calculation. U1 outputs the prediction noise for time step t+1. It should be understood that since time step t+1 directly reuses the intermediate features extracted by D1 and U2 at time step t, only the calculation process of U1 in U-Net needs to be executed at time step t+1, without executing the calculation process of other sampling modules, thereby reducing the amount of computation and improving the speed of visual content generation. Next, based on the prediction noise at time step t+1, the denoiser in the diffusion model is used to denoise the intermediate visual content (such as latent space visual content) output at time step t, thereby generating the intermediate visual content output at time step t+1.

[0135] Similarly, when executing at time step t+2, since t+2 is a non-critical time step, the intermediate features generated by D1 and U2 at the previous critical time step (i.e., time step t) can be directly retrieved from the cache and then input into U1 for calculation. U1 outputs the prediction noise for time step t+2. It should be understood that because time step t+2 directly reuses the intermediate features extracted by D1 and U2 at time step t, only the calculation process of U1 in U-Net needs to be executed at time step t+2, without the need for calculations from other sampling modules, thus reducing computational load and improving the speed of visual content generation. Next, based on the prediction noise at time step t+2, the denoiser in the diffusion model is used to denoise the intermediate visual content (such as latent space visual content) output at time step t+1, thereby generating the intermediate visual content output at time step t+2.

[0136] As shown in Figure 8, the calculation process of U1 executed at time step t+1 and the calculation process of U1 executed at time step t+2 can be executed in parallel, further improving the speed of visual content generation.

[0137] When execution reaches time step t+3, since time step t+3 is a critical time step, the complete network computation of U-Net will be performed at time step t+3, and the intermediate features generated by D1 and U2 at time step t+3 will be written to the cache, thus updating the cache content.

[0138] Optionally, the number of parallel executions mentioned above can be set by the user or left as a default value. A higher number of parallel executions results in better acceleration of visual content generation.

[0139] For example, assuming the parallelism is set to 3, then at most 3 non-critical time steps of partial network computation can be executed in parallel at one time.

[0140] S502: Generate first visual content.

[0141] Specifically, when the diffusion model completes the iterative calculation of the last time step out of the above N time steps, the intermediate visual content (latent space visual content) VAE output by the last time step is decoded by the decoder in the VAE, and then the first visual content in the pixel space is restored.

[0142] The first visual content can be an image or a video. The type of the first visual content is related to the type of diffusion model, which can be found in the previous section and will not be repeated here.

[0143] In summary, the visual content generation method provided in this application sets key and non-key time steps in the N time steps that the diffusion model needs to iterate. The intermediate features generated by the complete network computation corresponding to the key time step are reused in the non-key time steps. This eliminates the need to perform the complete network computation of the diffusion model in the non-key time steps; only a portion of the network computation is required. This reduces computational load and increases the speed of visual content generation (shortening the visual content generation latency), thereby improving the user experience. Furthermore, the intermediate features generated in one key time step can be reused in multiple non-key time steps, and the partial network computations corresponding to these multiple non-key time steps can be executed in parallel, further shortening the visual content generation latency and increasing the speed of visual content generation.

[0144] Please refer to Figure 9. This application embodiment also provides a visual content generation device 900, including a processing module 910.

[0145] The processing module 910 is used to: perform N time-step iterations on the diffusion model to generate the first visual content. The N time steps include key time steps and non-key time steps. Key time steps indicate the execution of the complete network computation of the diffusion model, while non-key time steps indicate the execution of partial network computations of the diffusion model. The intermediate features generated by the complete network computation corresponding to the key time steps are used as input to the partial network computation corresponding to the non-key time steps.

[0146] Optionally, the visual content generation device 900 further includes a determination module 920. Before the processing module 910 performs N time-step iterations on the diffusion model, the processing module 910 is further configured to: perform M time-step iterations on the diffusion model to generate second visual content. The determination module 920 is configured to: if the error between the intermediate features generated by the complete network computation corresponding to the first time step and the intermediate features generated by the complete network computation corresponding to the second time step is less than or equal to an error threshold, then determine the time step corresponding to the first time step in the N time steps as a critical time step, and determine the time step corresponding to the second time step in the N time steps as a non-critical time step. Each of the M time steps performs a complete network computation of the diffusion model, and the M time steps include the first time step and the second time step, with the first time step preceding the second time step.

[0147] Optionally, the visual content generation device 900 further includes a determination module 920. Before the processing module 910 performs N time-step iterations on the diffusion model, the processing module 910 is further configured to: perform M time-step iterations on the diffusion model to generate second visual content. The determination module 920 is configured to: determine M errors based on the intermediate features generated by the complete network computation corresponding to each of the M time steps, and then determine the time steps corresponding to K of the M errors in the N time steps as key time steps. Here, the M errors correspond one-to-one with the M time steps, and the error corresponding to each of the M time steps is the error between the intermediate features generated by the complete network computation corresponding to each time step and the intermediate features generated by the complete network computation corresponding to the next time step, where K is less than M.

[0148] Optionally, the above K errors are the K smallest among the M errors.

[0149] Optionally, the value of K mentioned above is specified by the user.

[0150] Optionally, there may be multiple non-critical time steps among the above N time steps. The intermediate features generated by the complete network computation corresponding to the critical time step are used as inputs for the partial network computations corresponding to multiple non-critical time steps. The partial network computations corresponding to multiple non-critical time steps are executed in parallel.

[0151] Optionally, the processing module 910 is specifically used to: when the iteration of N time steps reaches a critical time step, perform the complete network computation of the diffusion model and write the intermediate features generated by the complete network computation corresponding to the critical time step into the first storage area; when the iteration of N time steps reaches a non-critical time step, obtain the intermediate features generated by the complete network computation corresponding to the critical time step from the first storage area, and perform partial network computation of the diffusion model according to the obtained intermediate features generated by the complete network computation corresponding to the critical time step.

[0152] Optionally, the processing module 910 is specifically used to: add noise to the first intermediate visual content based on the noisy data to obtain the noisy first intermediate visual content; then, denoise the noisy first intermediate visual content based on the prediction noise and diffusion model of the non-critical time steps to generate the intermediate visual content output at the non-critical time steps. The output of the partial network computation corresponding to the non-critical time steps includes the prediction noise of the non-critical time steps, and the output of the complete network computation corresponding to the critical time steps includes the aforementioned first intermediate visual content.

[0153] Optionally, the intermediate features generated by the complete network computation at the key time steps are the outputs of the first downsampling module and the second upsampling module. The partial network computation at non-key time steps includes the computation of the first upsampling module. Specifically, the first downsampling module is the one with the largest output feature size among the multiple downsampling modules in the diffusion model, the first upsampling module is the one with the largest output feature size among the multiple upsampling modules in the diffusion model, and the second upsampling module is the one with the second largest output feature size among the multiple upsampling modules.

[0154] Optionally, the aforementioned first visual content can be an image or a video.

[0155] It should be noted that the visual content generation device 900 in Figure 9 can be used to execute the visual content generation method of the embodiment in Figure 5. For details, please refer to the above description, which will not be repeated here.

[0156] It should also be noted that the visual content generation device 900 in Figure 9 is only functionally and exemplaryly divided into a processing module 910 and a determining module 920. In reality, the visual content generation device 900 in Figure 9 can also contain more or fewer modules. For example, one of the above modules can be split into multiple functional modules, or two or more of the above modules can be merged into one functional module. Other functional modules can also be added to the visual content generation device 900 in Figure 9; this application does not limit this. Both the processing module 910 and the determining module 920 can be implemented in software or in hardware. For example, the implementation of the processing module 910 will be described below. Similarly, the implementation of the other modules can refer to the implementation of the processing module 910.

[0157] As an example of a software functional unit, processing module 910 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs. Similarly, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC. The interconnection between VPCs is achieved through the communication gateway.

[0158] As an example of a hardware functional unit, the processing module 910 may include at least one computing device, such as a server. Alternatively, the processing module 910 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0159] The processing module 910 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 910 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 910 includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0160] Referring to Figure 10, this application also provides a computing device 1000, including a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server, laptop computer, tablet computer, desktop computer, edge device, smartphone, etc., and this application does not specifically limit it, nor does it limit the number of processors and memories in the computing device 1000.

[0161] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 10, but this does not imply that there is only one bus or one type of bus. Bus 1002 can include pathways for transmitting information between various components of computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0162] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0163] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0164] The memory 1006 stores executable program code. The processor 1004 executes the executable program code to implement the functions of the processing module 910 and the determination module 920 in FIG9, respectively, thereby implementing the steps in the visual content generation method of the embodiment of FIG5 of this application.

[0165] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0166] As one possible implementation, the computing device 1000 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, which executes the steps in the visual content generation method of Figure 5. For simplicity, details are omitted here. The processor can be implemented using a CPU, or it can be implemented using computing devices or AI chips such as GPUs, DPUs, NPUs, XPUs, SoCs, offloading cards, or accelerator cards.

[0167] As one possible implementation, the computing device 1000 may include various types of processors 1004, meaning the computing device 1000 is a heterogeneous device. For example, the computing device 1000 may include a CPU and a GPU, and at least one of the processors 1004 may execute the operational steps in the visual content generation method of FIG5. For the sake of brevity, further details will not be provided here.

[0168] As shown in Figure 11, this application also provides a computing device cluster, which includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same instructions for implementing the method of Figure 5.

[0169] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for implementing the method of FIG5. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions for implementing the method of FIG5.

[0170] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, which are used to execute some functions of the visual content generation device 900 in FIG9. That is, the instructions stored in the memory 1006 in different computing devices 1000 can implement the functions of one or more modules in the processing module 910 and the determining module 920.

[0171] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 illustrates one possible implementation. As shown in Figure 12, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1006 in computing device 1000A stores instructions for executing the functions of processing module 910. Simultaneously, the memory 1006 in computing device 1000B stores instructions for executing the functions of determination module 920.

[0172] It should be understood that the functions of computing device 1000A shown in Figure 12 can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0173] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster shown in Figure 12. The difference is that the memory 1006 in one or more computing devices 1000 in this computing device cluster can store the same instructions for implementing the method of Figure 5 above.

[0174] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for implementing the method of FIG5. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions for implementing the method of FIG5.

[0175] This application also provides a chip system including a processor and a power supply circuit. The power supply circuit supplies power to the processor, which executes the operation steps in the method shown in Figure 5. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices such as a DPU, NPU, XPU, SoC, offloading card, or accelerator card.

[0176] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the operational steps in the visual content generation method shown in FIG5.

[0177] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the operational steps in the visual content generation method of FIG5.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating visual content, characterized in that, The method includes: The diffusion model is iterated over N time steps to generate the first visual content. The N time steps include key time steps and non-key time steps. The key time steps indicate the execution of the full network computation of the diffusion model, and the non-key time steps indicate the execution of a partial network computation of the diffusion model. The intermediate features generated by the full network computation corresponding to the key time step are used as the input to the partial network computation corresponding to the non-key time step.

2. The method according to claim 1, characterized in that, Before performing N time-step iterations on the diffusion model, the method further includes: The diffusion model is iterated for M time steps to generate second visual content, wherein each of the M time steps performs a full network computation of the diffusion model; If the error between the intermediate features generated by the complete network computation corresponding to the first time step and the intermediate features generated by the complete network computation corresponding to the second time step is less than or equal to the error threshold, then the time step corresponding to the first time step in the N time steps is determined as the critical time step, and the time step corresponding to the second time step in the N time steps is determined as the non-critical time step, wherein the M time steps include the first time step and the second time step, and the first time step is earlier than the second time step.

3. The method according to claim 1, characterized in that, Before performing N time-step iterations on the diffusion model, the method further includes: The diffusion model is iterated for M time steps to generate second visual content, wherein each of the M time steps performs a full network computation of the diffusion model; M errors are determined based on the intermediate features generated by the complete network computation corresponding to each of the M time steps. The M errors correspond one-to-one with the M time steps, and the error corresponding to each of the M time steps is the error between the intermediate features generated by the complete network computation corresponding to each time step and the intermediate features generated by the complete network computation corresponding to the next time step of each time step. The time steps corresponding to K errors out of the M errors are determined as the time steps corresponding to the N time steps, where K is less than M.

4. The method according to claim 3, characterized in that, The K errors are the K smallest of the M errors.

5. The method according to claim 3 or 4, characterized in that, The value of K is specified by the user.

6. The method according to any one of claims 1 to 5, characterized in that, The number of non-critical time steps in the N time steps is multiple. The intermediate features generated by the complete network computation corresponding to the critical time step are used as inputs for the partial network computations corresponding to multiple non-critical time steps. The partial network computations corresponding to multiple non-critical time steps are executed in parallel.

7. The method according to any one of claims 1 to 6, characterized in that, The process of performing N time-step iterations on the diffusion model includes: When the iteration of the N time steps reaches the key time step, the complete network computation of the diffusion model is performed, and the intermediate features generated by the complete network computation corresponding to the key time step are written into the first storage area. When the iteration of the N time steps reaches the non-critical time step, the intermediate features generated by the complete network computation corresponding to the critical time step are obtained from the first storage area, and the partial network computation of the diffusion model is performed according to the obtained intermediate features generated by the complete network computation corresponding to the critical time step.

8. The method according to any one of claims 1 to 7, characterized in that, The output of the partial network computation corresponding to the non-critical time step includes the prediction noise of the non-critical time step, and the output of the complete network computation corresponding to the critical time step includes the first intermediate visual content. The iteration of the diffusion model for N time steps includes: Noise is added to the first intermediate visual content based on the noise data to obtain the noisy first intermediate visual content. The first intermediate visual content after noise addition is denoised based on the predicted noise of the non-critical time step and the diffusion model to generate the intermediate visual content output by the non-critical time step.

9. The method according to any one of claims 1 to 8, characterized in that, The intermediate features generated by the complete network computation corresponding to the critical time step are output by the first downsampling module and the second upsampling module. The partial network computation corresponding to the non-critical time step includes the computation of the first upsampling module. The first downsampling module is the downsampling module with the largest output feature size among the multiple downsampling modules in the diffusion model. The first upsampling module is the upsampling module with the largest output feature size among the multiple upsampling modules in the diffusion model. The second upsampling module is the upsampling module with the second largest output feature size among the multiple upsampling modules.

10. The method according to any one of claims 1 to 9, characterized in that, The first visual content is an image or video.

11. A visual content generation device, characterized in that, include: The processing module is used to perform N time-step iterations on the diffusion model to generate first visual content. The N time steps include key time steps and non-key time steps. The key time steps indicate the execution of the complete network computation of the diffusion model, and the non-key time steps indicate the execution of partial network computation of the diffusion model. The intermediate features generated by the complete network computation corresponding to the key time step are used as the input to the partial network computation corresponding to the non-key time step.

12. The apparatus according to claim 11, characterized in that, The device also includes a determining module; The processing module is further configured to: perform M time-step iterations on the diffusion model to generate second visual content, wherein each of the M time-steps performs a complete network computation of the diffusion model; The determining module is used to: if the error between the intermediate features generated by the complete network computation corresponding to the first time step and the intermediate features generated by the complete network computation corresponding to the second time step is less than or equal to the error threshold, then determine the time step corresponding to the first time step in the N time steps as the critical time step, and determine the time step corresponding to the second time step in the N time steps as the non-critical time step, wherein the M time steps include the first time step and the second time step, and the first time step is earlier than the second time step.

13. The apparatus according to claim 11, characterized in that, The device also includes a determining module; The processing module is further configured to: perform M time-step iterations on the diffusion model to generate second visual content, wherein each of the M time-steps performs a complete network computation of the diffusion model; The determining module is used to: determine M errors based on the intermediate features generated by the complete network computation corresponding to each of the M time steps, wherein the M errors correspond one-to-one with the M time steps, and the error corresponding to each of the M time steps is the error between the intermediate features generated by the complete network computation corresponding to each time step and the intermediate features generated by the complete network computation corresponding to the next time step of each time step. The determining module is further configured to: determine the time steps corresponding to K errors out of the M errors as the key time steps among the N time steps, wherein K is less than M.

14. The apparatus according to claim 13, characterized in that, The K errors are the K smallest of the M errors.

15. The apparatus according to claim 13 or 14, characterized in that, The value of K is specified by the user.

16. The apparatus according to any one of claims 10 to 15, characterized in that, The number of non-critical time steps in the N time steps is multiple. The intermediate features generated by the complete network computation corresponding to the critical time step are used as inputs for the partial network computations corresponding to multiple non-critical time steps. The partial network computations corresponding to multiple non-critical time steps are executed in parallel.

17. The apparatus according to any one of claims 11 to 16, characterized in that, The processing module is specifically used for: When the iteration of the N time steps reaches the key time step, the complete network computation of the diffusion model is performed, and the intermediate features generated by the complete network computation corresponding to the key time step are written into the first storage area. When the iteration of the N time steps reaches the non-critical time step, the intermediate features generated by the complete network computation corresponding to the critical time step are obtained from the first storage area, and the partial network computation of the diffusion model is performed according to the obtained intermediate features generated by the complete network computation corresponding to the critical time step.

18. The apparatus according to any one of claims 11 to 17, characterized in that, The output of the partial network computation corresponding to the non-critical time step includes the prediction noise of the non-critical time step, and the output of the complete network computation corresponding to the critical time step includes the first intermediate visual content. The processing module is specifically used for: Noise is added to the first intermediate visual content based on the noise data to obtain the noisy first intermediate visual content. The first intermediate visual content after noise addition is denoised based on the predicted noise of the non-critical time step and the diffusion model to generate the intermediate visual content output by the non-critical time step.

19. The apparatus according to any one of claims 11 to 18, characterized in that, The intermediate features generated by the complete network computation corresponding to the critical time step are output by the first downsampling module and the second upsampling module. The partial network computation corresponding to the non-critical time step includes the computation of the first upsampling module. The first downsampling module is the downsampling module with the largest output feature size among the multiple downsampling modules in the diffusion model. The first upsampling module is the upsampling module with the largest output feature size among the multiple upsampling modules in the diffusion model. The second upsampling module is the upsampling module with the second largest output feature size among the multiple upsampling modules.

20. The apparatus according to any one of claims 11 to 19, characterized in that, The first visual content is an image or video.

21. A computing device, characterized in that, Including processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 10.

22. A chip system, characterized in that, The chip system includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to perform the method as described in any one of claims 1 to 10.

23. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 10.

Images