Video generation model acceleration method and system based on joint optimization

Abstract

The invention provides a video generation model acceleration method and system based on joint optimization, and the method comprises the steps: analyzing the feature difference between a current time step and an adjacent time step when the current time step is executed, and carrying out the dynamic feature multiplexing; based on the feature difference, analyzing the feature gradient amplitude of the current time step, and performing bit width quantization adjustment on different layers of the network of the video generation model at the current time step; and repeating the above steps, executing all time steps, and completing acceleration of the video generation model. Based on a joint optimization technology, the strategy of dynamic feature multiplexing, adaptive quantization and skipping of correlation calculation in non-key layer calculation is realized. Through the collaborative optimization mechanism, the speed-up ratio is remarkably improved, and the performance limitation faced by a single optimization technology is effectively broken through. In the video generation process, the method can dynamically sense and adapt to the spatio-temporal variation characteristics of the video content, so that the acceleration strategy can be adaptively adjusted according to the evolution of the video content.

Description

Technical Field

[0001] This application relates to the field of deep learning model acceleration technology in the field of artificial intelligence, and more specifically, to a method and system for accelerating video generation models based on joint optimization. Background Technology

[0002] Current mainstream video generation models employ architectures based on U-Net or Transformer, with Diffusion Transform (DiTs) models gradually becoming the mainstream technology due to their superior generation quality. However, DiTs models suffer from two major technical drawbacks: First, the computational complexity of the attention mechanism during model inference is quadratic with the sequence length, leading to significant delays when generating high-resolution videos. In practical tests, the Open-Sora model takes 130 seconds to generate 64 frames of 512×512 resolution video on an NVIDIA A800 graphics card, severely limiting its practical applications.

[0003] Existing acceleration techniques mainly fall into three independent categories: quantization techniques reduce computation by lowering parameter precision, but existing methods such as VidiTQ use static bit width allocation, which cannot adapt to differences in feature sensitivity at different time steps; caching mechanisms such as Delta-DiT reduce computation by reusing historical features, but their fixed refresh rate leads to untimely updates of keyframes; and pruning techniques such as TinyFusion remove redundant network structures, but do not consider the fluctuations in inter-layer correlation caused by dynamic changes in time steps. When applied individually, these techniques can only achieve a maximum speedup of 2-3 times, and the generated quality is unstable.

[0004] A search revealed a Chinese patent with patent application number 202510378954.4, which discloses a dynamic quantization method and system for AIGC models. The core of this method is to optimize the model through layer-by-layer quantization and result verification: first, the original AIGC model is run to obtain initial results; then, each network layer is traversed sequentially, and the parameters of each layer are quantized to obtain an intermediate model. The intermediate model is run, and the results are compared with the initial results in terms of structural similarity. The quantized states of network layers with structural similarity ≥ a predetermined threshold and quantized computational density ≤ original computational density are retained, ultimately yielding the quantized model. However, its quantization method is relatively simple, and the quantization effect has room for improvement. Summary of the Invention

[0005] To address one of the shortcomings of existing technologies, the purpose of this application is to provide a method and system for accelerating video generation models based on joint optimization.

[0006] A first aspect of this application provides a method for accelerating video generation models based on joint optimization, comprising:

[0007] When executing the current time step, the feature differences between the current time step and the adjacent time steps are analyzed, and the video generation model is dynamically reused.

[0008] Based on the aforementioned feature differences, the feature gradient magnitude at the current time step is analyzed, and bit-width quantization is adjusted for different layers of the video generation model network at the current time step.

[0009] For each time step of the video generation model, the above-mentioned dynamic feature reuse and bit width quantization adjustment are continuously executed iteratively, ultimately accelerating the inference of the entire video generation model.

[0010] Optionally, the step of analyzing the feature differences between the current time step and adjacent time steps during the execution of the current time step, and dynamically reusing features for the video generation model, includes:

[0011] Calculate the feature difference, i.e., the L2 norm distance D between the feature maps of the network layers at the current time step and those at adjacent time steps. t :

[0012]

[0013] L represents the total number of network layers. The feature map representing the l-th layer at time step t. Let be the gradient between the l-th layer video frames at time step t. The feature map represents the l-th layer at time step tk, where 1 ≤ k ≤ 5 and k is an integer.

[0014] Set a dynamic threshold τ, when D t When the time step >τ, the cache is updated using the full-precision calculation result; otherwise, the features of the l-th layer of the network at the current time step are reused from the cached features of the l-th layer of the network at the previous time step tk.

[0015] Optionally, the step of analyzing the feature gradient magnitude at the current time step based on the feature differences and performing bit-width quantization adjustment on different layers of the video generation model network at the current time step includes:

[0016] Based on the aforementioned feature differences, a spatiotemporal two-dimensional sensitivity is constructed;

[0017] Based on the aforementioned spatiotemporal two-dimensional sensitivity, the computational precision of the network layer of the video generation model is divided into three levels: 8-bit, 6-bit, and 4-bit.

[0018] Optionally, constructing a spatiotemporal two-dimensional sensitivity based on the feature differences includes:

[0019] The feature difference of the l-th layer at time step t is taken as the spatiotemporal two-dimensional sensitivity S(t,l) of the l-th layer at time step t.

[0020] Optionally, based on the spatiotemporal two-dimensional sensitivity, the computational precision of the network layer of the video generation model is divided into three levels: 8-bit, 6-bit, and 4-bit, including:

[0021] Setting Q 30 and Q 70 This is the threshold used to divide the sensitive region;

[0022] S(t,l)>Q 70 At that time, the l-th layer uses 8-bit quantization to preserve complete feature information;

[0023] Q 30 ≤S(t,l)≤Q 70 At that time, the l-th layer uses 6-bit quantization to accelerate the calculation while ensuring basic accuracy;

[0024] S(t,l) 30 At this time, 4-bit quantization is enabled in the l-th layer to maximize computational efficiency.

[0025] Optionally, it also includes skipping the computation process of non-critical layers by calculating the feature cosine similarity between network layers at each time step.

[0026] A second aspect of this application provides a video generation model acceleration system based on joint optimization, comprising:

[0027] Dynamic feature reuse module: When executing the current time step, analyze the feature differences between the current time step and adjacent time steps, and perform dynamic feature reuse on the video generation model;

[0028] Adaptive quantization module: Based on the feature differences, analyzes the feature gradient magnitude at the current time step, and performs bit-width quantization adjustment on different layers of the video generation model network at the current time step;

[0029] Iterative Loop Module: For each time step of the video generation model, the dynamic feature reuse module and adaptive quantization module are continuously executed iteratively to ultimately accelerate the inference of the entire video generation model.

[0030] A third aspect of this application provides a video generation method, comprising:

[0031] Input video description text into the Open-Sora 1.2 video generation model;

[0032] By employing any of the video generation model acceleration methods or systems based on joint optimization described above, a video corresponding to the video description text generated by the model is obtained.

[0033] ​A fourth aspect of this application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the aforementioned video generation model acceleration method based on joint optimization, or to run the aforementioned video generation model acceleration system based on joint optimization.

[0034] A fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the aforementioned video generation model acceleration method based on joint optimization, or to run the aforementioned video generation model acceleration system based on joint optimization.

[0035] The video generation model acceleration method proposed in this application, based on joint optimization, significantly improves the speedup ratio by employing a collaborative optimization mechanism of dynamic feature reuse and adaptive quantization adjustment, effectively overcoming the performance bottleneck encountered by single optimization techniques. During the video generation process, this method can dynamically detect and adapt to the spatiotemporal changes in video content, thereby enabling the acceleration strategy to adaptively adjust according to the development of the video content.

[0036] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0037] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0038] Figure 1 This is a flowchart illustrating a video generation model acceleration method based on joint optimization according to an exemplary embodiment;

[0039] Figure 2 This is a flowchart illustrating, according to an exemplary embodiment, a process of skipping unnecessary calculations.

[0040] Figure 3 This is a structural diagram of a video generation model acceleration system based on joint optimization, according to an exemplary embodiment. Detailed Implementation

[0041] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0042] In the description of the embodiments in this application, "multiple" means two or more, unless otherwise explicitly specified. In this application, unless otherwise explicitly specified and limited, the terms "installed," "connected," "linked," "fixed," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0043] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0044] Existing acceleration technologies mainly fall into three categories, each with its own drawbacks: quantization techniques (such as VidiTQ) reduce computation by lowering parameter precision, but their static bit width allocation cannot adapt to differences in feature sensitivity at different time steps; caching mechanisms (such as Delta-DiT) reuse historical features to reduce computation, but their fixed refresh rate can lead to untimely keyframe updates; pruning techniques (such as TinyFusion) remove redundant network structures, but they do not consider the fluctuations in inter-layer correlation caused by dynamic changes in time steps. When used individually, these techniques can only achieve a maximum speedup of 2-3 times, and the generated quality is unstable. Based on these problems, this application provides a video generation model acceleration method based on joint optimization to address the aforementioned issues.

[0045] Reference Figure 1 As shown, a method for accelerating video generation models based on joint optimization includes the following steps:

[0046] S100: When executing the current time step, analyze the feature differences between the current time step and adjacent time steps, and perform dynamic feature reuse on the video generation model;

[0047] S200, based on feature differences, analyzes the feature gradient magnitude at the current time step and performs bit-width quantization adjustment on different layers of the video generation model network at the current time step;

[0048] S300, for each time step of the video generation model, the above-mentioned dynamic feature reuse and bit width quantization adjustment are continuously executed iteratively to ultimately accelerate the inference of the entire video generation model.

[0049] The embodiments described above utilize a collaborative optimization mechanism of dynamic feature reuse and adaptive quantization adjustment, significantly improving the speedup ratio and effectively overcoming the performance bottleneck encountered by single optimization techniques. During the video generation process, this method can dynamically detect and adapt to the spatiotemporal changes in video content, thereby enabling the acceleration strategy to adaptively adjust according to the development of the video content.

[0050] During video generation, network features at adjacent time steps often exhibit high similarity. Traditional methods perform complete computation for each time step, resulting in a large number of repetitive feature extraction operations. Consequently, in resource-constrained scenarios such as edge computing, complete frame-by-frame feature extraction can lead to computational delays. To address this computational redundancy issue, in some specific embodiments of this application, in step S100, when executing the current time step, the feature differences between the current time step and adjacent time steps are analyzed, and dynamic feature reuse is performed. This can be achieved through the following steps:

[0051] S101, establish a dynamic cache management mechanism based on feature difference analysis. This is achieved by calculating the L2 norm distance D between feature maps of adjacent time steps in the network layers. t ,in

[0052]

[0053] L represents the total number of network layers. The feature map representing the l-th layer at time step t. Let be the gradient between the l-th layer video frames at time step t.

[0054] S102, Set the dynamic threshold τ, when D t If the cache update operation is performed at time τ, the historical cache features are reused. Specifically, the l-th layer of the network at time step t reuses the features of the l-th layer of the network at time step tk, where 1≤k≤5 and k is an integer.

[0055] The embodiments described above in this application leverage the high similarity of network features at adjacent time steps during video generation, reusing historical features to avoid performing complete calculations for each time step, reducing repetitive feature extraction operations, and lowering the computational load. In resource-constrained scenarios such as edge computing, dynamic reuse reduces the computational load by approximately 60%, reduces computational latency caused by frame-by-frame feature extraction, significantly improves system real-time performance, and allows video generation to operate more efficiently even in resource-limited environments.

[0056] The above embodiments calculate the feature sensitivity differences between different time steps of the video generation model. However, traditional uniform bit-width quantization (e.g., 8-bit throughout) cannot adapt to the feature sensitivity differences between different time steps of the video generation model. Therefore, in some specific embodiments of this application, in step S200, based on the feature differences, the feature gradient magnitude at the current time step is analyzed, and bit-width quantization is adjusted for different layers of the video generation model network at the current time step. This can be achieved in the following way:

[0057] S201, based on feature differences, constructs a spatiotemporal two-dimensional sensitivity;

[0058]

[0059] S202, based on spatiotemporal two-dimensional sensitivity, divides the computational precision of the network layer of the video generation model into three levels: 8-bit, 6-bit, and 4-bit.

[0060] Specifically, the S matrix reflects both temporal variation (temporal difference term) and spatial importance (gradient term) without requiring additional calculation. A larger value indicates greater sensitivity to the features of that time step and network layer (requiring high-precision processing).

[0061] A quantization strategy based on the spatiotemporal sensitivity matrix S(t,l) enables intelligent allocation of computing resources. The core of this mechanism is to divide the computational precision of the network layers into three levels by analyzing the numerical distribution of S(t,l):

[0062] In highly sensitive regions, S(t,l) is greater than Q. 70 At this time, 8-bit high-precision quantization is used to preserve complete feature information. This region usually corresponds to the fast-moving parts of the video sequence or the feature layer with rich spatial details, and has a low tolerance for quantization error.

[0063] In the sensitive region, S(t,l) is greater than or equal to Q. 30 And less than or equal to Q 70 At this time, 6-bit balanced quantization is used to accelerate the calculation while ensuring basic accuracy, which is suitable for feature regions with moderate degree of change.

[0064] In the low-sensitivity region, S(t,l) is less than Q. 30 At this time, 4-bit aggressive quantization is enabled to maximize computational efficiency.

[0065] The embodiments described above in this application rely on a spatiotemporal two-dimensional evaluation matrix to dynamically adjust the quantization bit width in both spatial and temporal dimensions. In the spatial dimension, a higher bit width is maintained for highly sensitive time steps to reduce accuracy loss; in the temporal dimension, a lower bit width is automatically switched for statically insensitive time steps to balance efficiency, effectively avoiding the problem of accuracy loss in static quantization.

[0066] For time-varying data such as video streams, this approach avoids the drawbacks of fixed quantization strategies. During periods of intense motion, it maintains a high bit width to prevent feature distortion; during periods of stillness, it automatically switches to a low bit width to avoid resource waste, significantly improving its adaptability to dynamic scenes.

[0067] In video generation neural networks, traditional methods perform indiscriminate computation on all layers within the same time step. To skip unnecessary computations, some specific embodiments of this application, such as... Figure 2 As shown, at each time step, when running the video generation model, the cosine similarity of features between real-time computation network layers is calculated:

[0068]

[0069] When the value is detected to be greater than a preset threshold, the calculation process for non-critical layers is automatically skipped. Generally, the threshold is set to 0.9.

[0070] Specifically, skipping the calculation process of non-critical layers means that for non-critical layers such as layer L2, the calculation results of layer L1 are directly used, or the calculation of layer L1 is skipped.

[0071] The embodiments described above in this application automatically skip non-critical layer calculations when the cosine similarity of features between network layers exceeds a preset threshold, without affecting the generation quality. This directly reduces unnecessary computational steps, lowers computational complexity, allows the model to complete calculations faster, and improves overall operating efficiency. The embodiments of this application focus on critical layer calculations, avoiding excessive computational resource allocation to layers with high relevance and limited contribution. Resources such as computing power are used more effectively for critical parts that significantly impact the generation results, resulting in more rational resource allocation and facilitating efficient model operation. In conjunction with the dynamic feature reuse stage (efficient feature reuse reduces computational load) and the adaptive quantization stage (precise quantization improves efficiency), a more comprehensive dynamic optimization framework is built by addressing feature reuse, quantization control, and computational path optimization. This comprehensively improves the overall system performance, resulting in superior model performance in terms of efficiency and resource utilization.

[0072] Dynamic feature reuse, adaptive quantization, and correlation calculation optimization each rely on different triggering conditions (L1 norm, gradient magnitude, and cosine similarity, respectively). Directly combining them may lead to temporal misalignment and error accumulation. For example, quantization may affect the accuracy of feature reuse, and low-bit quantization may compromise the precision of correlation calculation. To address this, some specific embodiments of this application employ joint optimization to ensure the three modules share the same optimization objective. Simultaneously, a sliding window calibration is introduced to dynamically adjust the update frequency of each module, thereby preventing error amplification.

[0073] All three techniques (dynamic feature reuse, adaptive quantization, and correlation calculation optimization) require dynamic adjustment of the computation process (including cache updates, bit width switching, layer skipping, etc.). However, due to limitations in hardware (such as GPU) computing power, frequent adjustments can easily lead to resource contention and latency. To address this, some specific implementations in this application decompose quantization, reuse, and computation path optimization into asynchronous tasks, reducing conflicts through staggered execution. Furthermore, they optimize the memory access mechanism, leveraging the shared memory cache reuse feature to reduce data transfer overhead.

[0074] Based on the same technical concept, other embodiments of this application provide a video generation model acceleration system 100 based on joint optimization, such as... Figure 3 As shown, it includes:

[0075] Dynamic feature reuse module 110: When executing the current time step, it analyzes the feature differences between the current time step and adjacent time steps and performs dynamic feature reuse on the video generation model;

[0076] Adaptive quantization module 120: Based on feature differences, analyzes the feature gradient magnitude at the current time step and performs bit-width quantization adjustment on different layers of the video generation model network at the current time step;

[0077] Iterative Loop Module: For each time step of the video generation model, the dynamic feature reuse module and adaptive quantization module are continuously executed iteratively to ultimately accelerate the inference of the entire video generation model.

[0078] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the video generation model acceleration method based on joint optimization in the above embodiments, and will not be repeated here.

[0079] Based on the same technical concept, other embodiments of this application also provide a video generation method, including:

[0080] Input video description text into the Open-Sora 1.2 video generation model;

[0081] The video generation model acceleration method based on joint optimization or the aforementioned video generation model acceleration system based on joint optimization is used to obtain the video generated by the model corresponding to the video description text.

[0082] In a specific application example, with an NVIDIA A800-80GB graphics card as the hardware platform and the input task being the generation of 64-frame 512×512 resolution video, the video generation model acceleration method based on joint optimization achieved significant technical effects: at the hardware level, the measured end-to-end inference speed reached 6.23 times faster (based on Open-Sora benchmark tests); at the quality level, the VBench metrics remained almost unchanged; and in terms of applicability, it can dynamically adapt to the generation requirements of different resolutions (256×256 to 1024×1024) and frame rates (24-60fps).

[0083] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-described video generation model acceleration method based on joint optimization, or to run the above-described video generation model acceleration system based on joint optimization.

[0084] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the program can be used to execute the above-described video generation model acceleration method based on joint optimization, or to run the above-described video generation model acceleration system based on joint optimization.

[0085] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDRSDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0086] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0087] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0088] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0089] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for accelerating video generation models based on joint optimization, characterized in that, include: When executing the current time step, the feature differences between the current time step and the adjacent time steps are analyzed, and the video generation model is dynamically reused. Based on the aforementioned feature differences, the feature gradient magnitude at the current time step is analyzed, and bit-width quantization is adjusted for different layers of the video generation model network at the current time step. For each time step of the video generation model, the above-mentioned dynamic feature reuse and bit width quantization adjustment are continuously executed iteratively, ultimately accelerating the inference of the entire video generation model.

2. The video generation model acceleration method based on joint optimization according to claim 1, characterized in that, The step of analyzing the feature differences between the current time step and adjacent time steps during the execution of the current time step, and dynamically reusing features for the video generation model, includes: Calculate the feature difference, i.e., the L2 norm distance D between the feature maps of the network layers at the current time step and those at adjacent time steps. t : L represents the total number of network layers. The feature map representing the l-th layer at time step t. Let be the gradient between the l-th layer video frames at time step t. The feature map representing the l-th layer at time step tk; Set a dynamic threshold τ, when D t When the value is greater than τ, the cache is updated using the full-precision calculation result; otherwise, the features of the l-th layer of the network at the current time step are reused from the cached features of the l-th layer of the network at the previous tk time step, where 1≤k≤5 and k is an integer.

3. The video generation model acceleration method based on joint optimization according to claim 1, characterized in that, The step of analyzing the feature gradient magnitude at the current time step based on the feature differences and adjusting the bit width quantization of different layers of the video generation model network at the current time step includes: Based on the aforementioned feature differences, a spatiotemporal two-dimensional sensitivity is constructed; Based on the aforementioned spatiotemporal two-dimensional sensitivity, the computational precision of the network layer of the video generation model is divided into three levels: 8-bit, 6-bit, and 4-bit.

4. The video generation model acceleration method based on joint optimization according to claim 3, characterized in that, The construction of spatiotemporal two-dimensional sensitivity based on the aforementioned feature differences includes: The feature difference of the l-th layer at time step t is taken as the spatiotemporal two-dimensional sensitivity S(t,l) of the l-th layer at time step t.

5. The video generation model acceleration method based on joint optimization according to claim 3, characterized in that, Based on the aforementioned spatiotemporal two-dimensional sensitivity, the computational precision of the network layers of the video generation model is divided into three levels: 8-bit, 6-bit, and 4-bit, including: Setting Q 30 and Q 70 This is the threshold used to divide the sensitive region; S(t,l)>Q 70 At that time, the l-th layer uses 8-bit quantization to preserve complete feature information; Q 30 ≤S(t,l)≤Q 70 At that time, the l-th layer uses 6-bit quantization to accelerate the calculation while ensuring basic accuracy; S(t,l) 30 At this time, 4-bit quantization is enabled in the l-th layer to maximize computational efficiency.​ 6. The video generation model acceleration method based on joint optimization according to claim 1, characterized in that, It also includes skipping the computation process of non-critical layers by calculating the feature cosine similarity between network layers at each time step.

7. A video generation model acceleration system based on joint optimization, characterized in that, include: Dynamic feature reuse module: When executing the current time step, analyze the feature differences between the current time step and adjacent time steps, and perform dynamic feature reuse on the video generation model; Adaptive quantization module: Based on the feature differences, analyzes the feature gradient magnitude at the current time step, and performs bit-width quantization adjustment on different layers of the video generation model network at the current time step; Iterative Loop Module: For each time step of the video generation model, the dynamic feature reuse module and adaptive quantization module are continuously executed iteratively to ultimately accelerate the inference of the entire video generation model.

8. A video generation method, characterized in that, include: Input video description text into the Open-Sora 1.2 video generation model; Using the video generation model acceleration method based on joint optimization as described in any one of claims 1-6 or the video generation model acceleration system based on joint optimization as described in claim 7, a video generated by the model corresponding to the video description text is obtained.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-6 and 8, or to run the system of claim 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-6 and 8, or to run the system of claim 7.

Images