Video generation acceleration method, electronic equipment, computer storage medium and program product
By allocating fixed denoising time steps to multiple graphics processors and processing the denoising steps of the diffusion model in parallel, the problem of slow video generation speed of the diffusion model is solved, and efficient video generation and real-time streaming applications are realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-29
- Publication Date
- 2026-04-14
AI Technical Summary
Existing diffusion models suffer from low generation speed and efficiency in video generation due to the sequential dependence of multiple denoising steps.
By allocating fixed denoising time steps to multiple graphics processors and performing denoising generation processing in parallel, the process is transformed into an asynchronous spatial pipeline, decoupling the inference throughput of the diffusion model.
It improves the speed and efficiency of video generation, enables real-time streaming video generation, and reduces the complexity of matching GPU with time steps.
Smart Images
Figure CN121865050A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation acceleration method, an electronic device, a computer storage medium, and a computer program product. Background Technology
[0002] AI (Artificial Intelligence) based video generation is a technology that uses artificial intelligence algorithms to automatically create new video content by learning from large amounts of video data. Among AI-based video generation technologies, using diffusion models is one of the more commonly used methods.
[0003] Diffusion models are a type of image generation model that learns the information attenuation caused by noise and then uses the learned pattern to generate images. Currently, streaming video generation can be achieved with the help of large-scale (e.g., 14 billion parameters) diffusion models. However, because the multiple denoising steps of a diffusion model are sequentially dependent, multiple denoising steps are typically executed sequentially by a single processor, which severely impacts the speed and efficiency of video generation. Summary of the Invention
[0004] In view of this, embodiments of this application provide a video generation acceleration solution to at least partially solve the above-mentioned problems.
[0005] According to a first aspect of the embodiments of this application, a video generation acceleration method is provided, comprising: determining a plurality of graphics processors to be used for a video to be generated, wherein each graphics processor is assigned to perform a fixed denoising time step of a diffusion model, and the fixed denoising time steps corresponding to different graphics processors are different; acquiring a plurality of noise images for generating the video and an audio segment corresponding to each noise image, wherein the audio segments corresponding to different noise images are different; and performing denoising generation processing in parallel according to the order of the fixed denoising time steps corresponding to each of the plurality of graphics processors based on the plurality of noise images and the audio segments corresponding to the plurality of noise images, to obtain video frame images corresponding to the plurality of noise images.
[0006] According to a second aspect of the present application, an electronic device is provided, comprising: a plurality of graphics processors, an instruction chip, a memory, a communication interface, and a communication bus, wherein the plurality of graphics processors, the instruction chip, the memory, and the communication interface communicate with each other through the communication bus; the instruction chip is configured to allocate fixed time steps of a diffusion model to the plurality of graphics processors respectively according to the method described in the first aspect, so that the plurality of graphics processors perform denoising generation processing in parallel based on a plurality of noise images stored in the memory and an audio segment corresponding to each noise image, to obtain video frame images corresponding to the plurality of noise images.
[0007] According to a third aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.
[0008] According to a fourth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform an operation corresponding to the method described in the first aspect.
[0009] According to the solution provided in the embodiments of this application, when using a diffusion model to generate video, the traditional serial sequential denoising generation process is transformed into an asynchronous spatial pipeline by using multiple graphics processors to process in parallel according to a fixed denoising time step. This decouples the inference throughput of the diffusion model from the sum of all diffusion denoising generation steps to the propagation level of a single denoising generation step, effectively improving the speed and efficiency of video generation. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0011] Figure 1 This is a schematic diagram of a diffusion model for an existing technology. Figure 2A This is a flowchart illustrating the steps of a video generation acceleration method according to an embodiment of this application. Figure 2B for Figure 2A A schematic diagram of a scenario example in the illustrated embodiment; Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0012] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.
[0013] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.
[0014] To facilitate understanding of the solutions in the embodiments of this application, the diffusion model will be briefly and exemplaryly described below.
[0015] The diffusion model is a type of image generation model that learns the information attenuation caused by noise and then uses the learned pattern to generate an image. An illustration of this process is shown below. Figure 1 As shown, the diffusion model utilizes image noise addition in the forward process and image denoising in the reverse process, using the difference between the denoised image and the original image as the basis for training the model. In the inference phase of the model, the reverse process of the diffusion model can be directly used to generate the image.
[0016] For diffusion models, such as Figure 1 As shown, model training can be achieved using image samples, including: in the forward process (such as... Figure 1 In the forward process (from right to left), multiple diffusion noise-adding layers are used to add noise to the image samples to generate noisy image samples; in the reverse process (e.g., ... Figure 1 The process proceeds from left to right (reverse process). Then, the noisy image samples generated in the forward process are denoised to produce a denoised image. Based on the difference between the denoised image and the original image samples, the diffusion model is trained. For example, in the training phase of the diffusion model, its forward process gradually adds noise to data (such as clear image data) through diffusion, generating noisy data (such as noisy images). Figure 1 As shown, noise needs to be added to the image output from the previous diffusion noise layer according to preset parameters, so as to base the diffusion noise layer. Noise is added to the output image to generate a diffusion noise layer. The image shows the process illustrated in the diagram. The process involves a forward process, where diffusion noise is used to generate a noisy image sample (as shown in the left image of the figure). Then, based on this noisy image sample, a reverse process is used to progressively denoise and output the original image (as shown in the right image of the figure), thus creating a denoising layer. Denoising the output image and generating a denoising layer The image is used as an example; the process is illustrated in the diagram. .
[0017] After training, during the inference phase, image generation can be performed directly based on the inverse process of the diffusion model. At this stage, the input image can be a purely noisy image. In other words, during the inference phase of the diffusion model, image generation can be achieved using a purely noisy image. This image generation process continues sequentially to achieve video generation.
[0018] In the diffusion model, the number of steps in the forward and reverse processes is defined by the subscript t, and there is a predefined total number of steps T (exemplarily 1000). At t=0, a sampled real image is obtained; at t=T, it is approximately a purely noisy image. During the inference phase, the diffusion model generates a normal image, such as an RGB image, from a noisy image by taking time steps from T to 0.
[0019] Based on the principles of the above diffusion model, the video generation acceleration method provided in this application will be described below through examples.
[0020] Reference Figure 2A The diagram illustrates a flowchart of a video generation acceleration method according to an embodiment of this application.
[0021] The video generation acceleration method in this embodiment includes the following steps: Step S202: Determine the multiple graphics processors to be used for the video to be generated.
[0022] In this embodiment of the application, the device for video generation is equipped with multiple GPUs (Graphics Processing Units), which can work together in parallel to accelerate video generation.
[0023] Each graphics processing unit (GPU) is assigned to execute one or more fixed time steps of the diffusion model; the fixed time steps differ for different GPUs. For example... Figure 1 As shown, in the inference phase of the diffusion model, there can be T time steps, each of which can be represented as... , where 0≤ ≤T-1.
[0024] In one alternative approach, the number of GPUs can be equal to T. In this case, each time step is assigned to one GPU, and T time steps are assigned to T GPUs. For example, assuming there are 5 time steps and 5 GPUs, each GPU is assigned one time step. For instance, GPU0 is assigned to time step T. GPU1 was allocated to the time step GPU2 was allocated to the time step. GPU3 was allocated to the time step. GPU4 was allocated to the time step This approach not only enables faster video generation but also significantly reduces the processing complexity of matching GPUs and time steps.
[0025] In another alternative approach, the number of GPUs can be less than T. In this case, the GPUs can be executed sequentially according to the time steps. First, a portion of the time steps can be executed by the GPUs. After the portion of the time steps is completed, the remaining time steps can be executed using the same GPUs (with the number of remaining time steps equal to the number of GPUs). Alternatively, after the portion of the time steps is completed, a portion of the GPUs can be executed (with the number of remaining time steps less than the number of GPUs). Or, a combination of the above two methods can be used until all time steps are executed (with the number of remaining time steps greater than the number of GPUs). This approach can save on the number of GPUs and reduce the implementation cost.
[0026] In another alternative approach, if the number of GPUs is less than T, a single GPU can be responsible for multiple consecutive time steps. For example, GPU0 can handle time steps 0 and 1, GPU1 can handle time steps 2 and 3, and so on. This approach not only saves on implementation costs but also simplifies resource matching and improves the efficiency of matching GPUs with time steps.
[0027] Step S204: Obtain multiple noise images for generating the video and the corresponding audio segment for each noise image.
[0028] The solution provided in this application can be used in scenarios where video is generated from audio. Since video consists of multiple video frames, it is necessary to segment the audio data into multiple different audio segments to combine with noise images to generate video frame images. Therefore, in this scenario, video generation can first obtain multiple noise images for video generation and the corresponding audio segment for each noise image.
[0029] In one alternative approach, audio data for video generation can be acquired first, and then divided into multiple non-overlapping audio segments according to a preset time length and chronological order. These audio segments are then assigned to corresponding pure noise images, allowing multiple graphics processors to process the pure noise images in chronological order of the audio segments. Different noise images correspond to different audio segments, and the preset time length can be flexibly set by those skilled in the art according to actual needs. Since the video formed from video frame images needs to match the audio data, subsequent processing of noise images must be based on the audio segment order. Pure noise images are readily available; this approach can, to some extent, reduce the cost of video generation and improve its efficiency.
[0030] Step S206: Based on multiple noisy images and the audio segments corresponding to each of the multiple noisy images, denoising generation processing is performed in parallel according to the order of the fixed denoising time steps corresponding to each of the multiple graphics processors, so as to obtain video frame images corresponding to the multiple noisy images.
[0031] As mentioned earlier, for multiple time steps between a noisy image and a video frame image, each time step is assigned to a GPU for execution. Each GPU executes its assigned time step in a fixed manner, and the time steps are executed in parallel among multiple GPUs.
[0032] For diffusion models, after denoising and image generation, the denoised image still needs to be decoded to generate the final image, such as an RGB image. Therefore, one feasible approach is to merge the image decoding time step with the last denoising time step, with the GPU executing the last denoising time step performing the image decoding time step. Because multiple denoising time steps are executed in parallel, the overall speed and efficiency of video generation can still be significantly improved even with the image decoding time step merged.
[0033] However, to prevent the video generation pipeline from becoming a bottleneck due to multiple GPUs and to further improve the speed and efficiency of video generation, in one optional approach, the aforementioned multiple GPUs include a dedicated GPU for image decoding. That is, image decoding is offloaded to an additional dedicated GPU, which receives the clean latent representation from the previous GPU and outputs synchronized video frame images. In this case, performing denoising and generation processes in parallel according to the fixed denoising time steps corresponding to each of the multiple GPUs to obtain video frame images corresponding to multiple noisy images can include: performing denoising and generation processes in parallel according to the fixed denoising time steps and image decoding time steps corresponding to each of the multiple GPUs to obtain video frame images corresponding to multiple noisy images. This avoids decoding becoming a bottleneck and ensures high end-to-end efficiency.
[0034] Furthermore, in one feasible approach, the aforementioned parallel processing of the GPU can be divided into two stages: a warm-up stage and a fully pipelined streaming stage. The warm-up stage is used to quickly bring multiple GPUs into processing mode, i.e., denoising generation or image decoding. The fully pipelined streaming stage is used to execute multiple time steps in parallel after all GPUs are in processing mode to achieve video generation.
[0035] During the warm-up phase, the first noisy image from a set of multiple noisy images and its corresponding audio segment can be processed across multiple GPUs, ensuring that all GPUs are in a processing state. Since the processing involves the first noisy image, the sampling steps are few, and this phase is completed quickly.
[0036] After the warm-up phase, once all GPUs are in processing mode, the process enters the fully pipelined streaming phase. This phase then executes denoising and generation operations in parallel, based on multiple noisy images and their corresponding audio segments, following the fixed denoising time steps assigned to each GPU. In this phase, each GPU repeatedly executes its assigned denoising time step, passing the denoised latent vector to the next GPU in the next time step and immediately processing the next noisy image, thus achieving maximum parallel throughput.
[0037] From an image perspective, for each noisy image and its corresponding audio segment, the denoising generation result of the previous GPU is sent to the next GPU in the order of the fixed denoising time step and image decoding time step corresponding to each of the multiple GPUs, so that the next GPU can perform the corresponding fixed denoising time step or image decoding time step processing.
[0038] From the GPU's perspective, for each GPU, at the same time, based on the audio segments of the historical images and the corresponding noisy images obtained by the GPU, the current image is denoised and generated according to the fixed denoising time step corresponding to the GPU, thus generating the denoising generation result.
[0039] The following, combined with Figure 2B The above process will be illustrated by example.
[0040] Figure 2B In the diagram, the horizontal axis represents the runtime, and the vertical axis represents the number of GPUs and the time steps each GPU needs to process. Figure 2B In the diagram, multiple time steps are simply illustrated in the order of execution. There are five time steps in total, among which, These are all noise reduction time steps. This is the image decoding time step. Correspondingly, This is a fixed denoising time step for GPU0, and it is always executed by GPU0. This is a fixed denoising time step for GPU1, and it is always executed by GPU1. This is a fixed denoising time step for GPU2, and is always executed by GPU2. This is a fixed denoising time step for GPU3, and it is always executed by GPU3. This is a fixed image decoding time step for GPU4, which is always executed by GPU4.
[0041] From an image perspective, Figure 2B Noisy images For example, it first goes through GPU0. The denoising generation process at the denoising time step yields... Denoising image And send it to GPU1; GPU1 receives it. After that, proceed The denoising generation process at the denoising time step yields... Denoising image And send it to GPU2; GPU2 receives it. After that, proceed The denoising generation process at the denoising time step yields... Denoising image And send it to GPU3; GPU3 receives it. After that, proceed The denoising generation process at the denoising time step yields... Denoising image And send it to GPU4; GPU4 receives it. After that, proceed The image decoding time step is processed to obtain the final RGB image. The processing of other noisy images is similar.
[0042] Meanwhile, for each GPU, such as GPU0, it always executes Denoising generation processing at the denoising time step. For example... Figure 2B As shown, when it is applied to a noisy image conduct The denoising generation process is performed at each denoising time step, and the denoised image is generated. After being sent to GPU1, the noisy image will be processed. conduct The denoising generation process is performed at each denoising time step, and the denoised image is generated. The image is sent to GPU1 for processing; then, GPU0 processes the noisy image. conduct The denoising generation process is performed at each denoising time step, and the denoised image is generated. Send it to GPU1. And so on, until the last noisy image used to generate the video is processed.
[0043] Depend on Figure 2B As can be seen, at some point after all GPUs are in the processing state, such as Figure 2B As shown in the dashed box, each GPU processes data based on its own time step, with multiple GPUs executing in parallel. As also shown in the dashed box, at any given moment, GPU0 processes the noisy image... conduct The denoising generation process at the denoising time step involves GPU1 processing the denoised image sent from GPUO. conduct The denoising generation process at the denoising time step involves GPU2 processing the denoised image sent from GPU1. conduct The denoising generation process at the denoising time step involves GPU3 processing the denoised image sent from GPU2. conduct The denoising generation process occurs at each denoising time step, while GPU4 processes the denoised image sent from GPU3. conduct Decoding processing of image decoding time steps.
[0044] Furthermore, unlike the conventional approach of using the previously decoded image as a constraint, in this embodiment, each GPU, when processing each image, uses historical images (e.g., four) of preset data processed by that GPU and the audio segments of the corresponding noise images as constraints for denoising and generation. Figure 2B Taking GPU1 as an example, it processes the current image... conduct During the denoising generation process at the denoising time step, the image obtained from the previous denoising generation process is used. and ,as well as Corresponding noise image The corresponding audio segment serves as a constraint condition for... The denoising generation process is performed at the denoising time step. This breaks through the inherent autoregressive denoising bottleneck of the diffusion model and achieves predictable low latency. Experiments show that 20 frames per second real-time streaming performance can be guaranteed using 5 H100 GPUs.
[0045] As can be seen from the above process, by allocating a fixed time step to each GPU and distributing T time steps to T devices, each GPU repeatedly executes the transformation at its assigned time step, transforming the original sequential diffusion chain of the diffusion model into an asynchronous spatial pipeline. Therefore, the throughput of video generation by the diffusion model is determined by a single denoising forward pass of the diffusion model, rather than the sum of all time steps. This effectively eliminates the sampling bottleneck during diffusion model inference, enabling real-time streaming generation of over 20 frames per second while maintaining the scale and quality advantages of multi-step diffusion models, effectively improving the speed and efficiency of video generation.
[0046] The above process is divided into two phases: a warm-up phase and a fully pipelined streaming phase. In the warm-up phase, the first noisy image is processed across all time steps to fill the processing pipeline formed by multiple GPUs. Since this phase involves fewer sampling steps, it is quick to complete. Once the warm-up phase has filled the pipeline with processing data from multiple GPUs, the system enters the fully pipelined streaming phase: each GPU repeatedly executes its assigned fixed time steps, passes latent features to the next GPU, and then immediately processes the next image, thereby achieving maximum parallel throughput. This allows the system to start quickly and continuously output video frames at a high frame rate, effectively supporting real-time streaming applications.
[0047] Once the video frame images are obtained through the above process, a video can be generated based on the video frame images.
[0048] Furthermore, optionally, for the multiple GPUs in this embodiment, each GPU has an independent cache for storing key-value (KV) values obtained by the GPU during image denoising and generation. After obtaining a preset number of KV values for each image, the obtained KV values replace the original KV values stored in the cache. Therefore, this cache can also be called a "rolling KV cache" to distinguish it from traditional caches. In an optional approach, the KV values in the cache can be dynamically updated using a sliding window mechanism, thereby reducing redundant calculations, lowering memory usage, and supporting longer video generation.
[0049] In traditional schemes without key-value (KV) caching, taking the Transformer architecture as an example, its self-attention mechanism needs to calculate the association between the current token and all historical tokens (i.e., attention score). Traditional methods require recalculating the key and value vectors of all historical tokens each time a new token is generated, resulting in a quadratic increase in computational complexity (O(T²)) with the sequence length. However, using the rolling KV caching in this embodiment, the computational complexity can be reduced to linear (O(T)) by storing historical KV vectors. For example, when a GPU performs denoising on the 10th image, it only needs to read the historical KV vectors of the 6th-9th denoised images stored in the rolling KV caching, calculate the new KV vector for the current image, and append it to the cache.
[0050] When using a sliding window mechanism to dynamically update key-value pairs, setting the sliding window size (e.g., caching the key-value vectors of the four most recent denoised images) and discarding the oldest key-value vectors while retaining the newest ones when a new image is generated can avoid the memory explosion problem caused by the infinite growth of the cache.
[0051] Furthermore, employing a rolling key-value (KV) cache can reduce the memory usage associated with repeatedly calculating historical KV vectors. For example, when generating a 4-minute video, the rolling KV cache can prevent memory usage from growing exponentially with the sequence length, making long video generation possible. Experimental data also shows that using a rolling KV cache significantly reduces redundant computations during video generation, improving generation speed and helping to stabilize generation quality, avoiding inconsistencies introduced by parallelization.
[0052] As can be seen, through the embodiments of this application, when using the diffusion model to generate video, the traditional serial sequential denoising generation process is transformed into an asynchronous spatial pipeline by using multiple graphics processors to process in parallel according to a fixed denoising time step. This decouples the inference throughput of the diffusion model from the sum of all diffusion denoising generation steps to the single denoising generation propagation level, effectively improving the speed and efficiency of video generation.
[0053] Reference Figure 3 The diagram illustrates a structural schematic of an electronic device according to Embodiment 5 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device. This electronic device can be used for video generation.
[0054] like Figure 3As shown, the electronic device may include: an instruction chip (a chip that can communicate with the GPU and send instructions, including but not limited to a CPU) 302, multiple GPUs 304, memory 306, a communication bus 308, and a communication interface 310.
[0055] in: The instruction chip 302, multiple GPUs 304, memory 306, and communication interface 310 communicate with each other through communication bus 308.
[0056] Communication interface 310 is used to communicate with other electronic devices or servers.
[0057] The instruction chip 302 is used to execute program 312, which may include computer instructions to perform the relevant steps in the above-described video generation method embodiments.
[0058] Memory 306 is used to store program 312, audio data for generating video, noise images, audio segments allocated to each noise image after segmenting the audio data, the correspondence between noise images and audio segments, etc. Memory 306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0059] The instruction chip 302 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application, etc. When video generation is required, the instruction chip first determines multiple GPUs 304 for video generation according to the program 312 in the memory 306; and allocates a fixed time step to each GPU according to the time step of the diffusion model to be used, so that the GPU repeatedly executes the time step during the video generation process; then, it retrieves multiple noise images and the corresponding audio segments from the memory 306; then, using the determined multiple GPUs 304, it performs denoising generation processing and image decoding processing in parallel based on the noise images and the corresponding audio segments to obtain video frame images corresponding to the multiple noise images. Based on these video frame images, the required video can be generated.
[0060] The specific implementation of each step in procedure 312 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.
[0061] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.
[0062] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the video generation methods in the above-described multiple method embodiments.
[0063] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, audio data used for generating videos, stored data, and displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0064] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.
[0065] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.
[0066] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.
[0067] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.
Claims
1. A method for accelerating video generation, comprising: Multiple graphics processors are determined for the video to be generated, where each graphics processor is assigned a fixed denoising time step to perform the diffusion model, and the fixed denoising time steps are different for different graphics processors; Obtain multiple noise images for generating the video and the corresponding audio segment for each noise image, wherein different noise images correspond to different audio segments; Based on multiple noisy images and their corresponding audio segments, denoising generation processes are performed in parallel according to the fixed denoising time steps of each of the multiple graphics processors to obtain video frame images corresponding to the multiple noisy images.
2. The method according to claim 1, wherein, The plurality of graphics processors includes one graphics processor for image decoding; The step of performing denoising generation processing in parallel according to the fixed denoising time steps corresponding to each of the multiple graphics processors to obtain video frame images corresponding to the multiple noise images includes: The denoising generation process is performed in parallel according to the fixed denoising time step and image decoding time step corresponding to each of the multiple graphics processors, to obtain video frame images corresponding to the multiple noise images.
3. The method according to claim 2, wherein, The denoising generation process, based on multiple noisy images and their corresponding audio segments, is performed in parallel according to the fixed denoising time steps corresponding to each of the multiple graphics processors, including: For each noisy image and the corresponding audio segment, the denoising generation result of the previous graphics processor is sent to the next graphics processor in the order of the fixed denoising time step and the image decoding time step corresponding to each of the multiple graphics processors, so that the next graphics processor can perform the corresponding fixed denoising time step or image decoding time step processing.
4. The method according to claim 3, wherein, For each graphics processor, at the same time, based on the audio segments of the historical image and the corresponding noise image obtained by the current graphics processor, the current image is denoised and generated according to the fixed denoising time step corresponding to the current graphics processor, and a denoising and generation result is generated.
5. The method according to any one of claims 1-4, wherein, Before performing denoising generation processing in parallel based on multiple noise images and their corresponding audio segments, according to the order of fixed denoising time steps corresponding to each of the multiple graphics processors, the method further includes: The first noise image and its corresponding audio segment from the plurality of noise images are processed in the plurality of graphics processors so that all of the plurality of graphics processors are in a processing state. After all the graphics processors are in processing mode, the operation of performing denoising generation processing in parallel, based on the multiple noise images and the audio segments corresponding to the multiple noise images, and in the order of the fixed denoising time steps corresponding to each of the multiple graphics processors, is executed.
6. The method according to any one of claims 1-4, wherein, Each of the plurality of graphics processors has an independent cache for storing KV values obtained by the graphics processor during image denoising and generation. Furthermore, after obtaining the KV values corresponding to a preset number of images, the obtained KV values are used to replace the original KV values stored in the cache.
7. The method according to any one of claims 1-4, wherein, The step of acquiring multiple noise images for generating the video and the corresponding audio segment for each noise image includes: Acquire audio data for generating video, and divide the audio data into multiple non-overlapping audio segments according to a preset time length in chronological order; The multiple audio segments are respectively assigned to the corresponding multiple pure noise images, so that the multiple graphics processors process the multiple pure noise images in the time sequence of the multiple audio segments.
8. An electronic device, comprising: Multiple graphics processors, instruction chips, memory, communication interfaces, and communication buses are provided, and the multiple graphics processors, instruction chips, memory, and communication interfaces communicate with each other through the communication bus; The instruction chip is configured to allocate fixed time steps of a diffusion model to the plurality of graphics processors according to the method of any one of claims 1-7, so that the plurality of graphics processors perform denoising generation processing in parallel based on the plurality of noise images stored in the memory and the audio segment corresponding to each noise image, so as to obtain video frame images corresponding to the plurality of noise images.
9. A computer storage medium having a computer program stored thereon, which, when executed by an instruction chip, implements the method as described in any one of claims 1-7.
10. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to any one of the methods described in claims 1-7.