A method, system and related devices for determining a data multiplexing strategy for a diffusion model

Abstract

The application provides a method, system and related equipment for determining a data multiplexing strategy of a diffusion model. The method comprises: a data processing system obtaining a first data multiplexing strategy, which is used to indicate that the diffusion model caches output data of a plurality of network layers at at least one time step, and indicates that the diffusion model multiplexes the cached output data of the plurality of network layers; in a process in which the diffusion model performs inference according to a first input, the data processing system caches and multiplexes the output data of the plurality of network layers according to the first data multiplexing strategy, completes the inference process of the diffusion model, and then optimizes the first data multiplexing strategy according to a reinforcement learning method and the output data of the plurality of network layers, to determine an optimized data multiplexing strategy, which is used to indicate that the output data of the network layers is cached and multiplexed in the inference process of the diffusion model. Through the above method, the efficiency of determining the data multiplexing strategy of the diffusion model can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, system, and related equipment for determining a data reuse strategy for a diffusion model. Background Technology

[0002] Diffusion models are artificial intelligence (AI) models that generate images or videos from text. During inference, diffusion models involve significant redundant computations. For example, some network layers in a diffusion model exhibit high similarity in their output features at adjacent time steps. Therefore, during inference, caching the output features of some network layers from the previous time step can reduce computational load and improve inference efficiency by reusing them in the next time step. However, current methods rely on manually specifying a data reuse strategy to instruct which network layers at which time steps to cache and reuse data. This method requires users to continuously modify and test different data reuse strategies to determine the one that best suits their needs, resulting in low efficiency. Therefore, determining a suitable data reuse strategy for diffusion models is a critical technical problem that needs to be solved to improve their inference efficiency. Summary of the Invention

[0003] This application provides a method, system, and related equipment for determining a data reuse strategy for a diffusion model, which can improve the efficiency of determining the data reuse strategy for a diffusion model.

[0004] In a first aspect, this application provides a method for determining a data reuse strategy for a diffusion model. The method includes: a data processing system acquiring a first data reuse strategy, the first data reuse strategy being used to instruct, during the inference process of the diffusion model, to cache the output data of multiple network layers of the diffusion model at at least one time step, and to instruct the diffusion model to reuse the cached output data of the multiple network layers; the data processing system, during the inference process of the diffusion model based on a first input, caching and reusing the output data of the aforementioned multiple network layers according to the first data reuse strategy to complete the inference process of the diffusion model; and then the data processing system optimizing the first data reuse strategy based on a reinforcement learning method and the output data of the multiple network layers to determine an optimized data reuse strategy, the optimized data reuse strategy being used to instruct the caching and reuse of the output data of the network layers during the inference process of the diffusion model.

[0005] The inference process of a diffusion model is a progressive denoising process. The diffusion model generates the final output by progressively denoising across multiple time steps. Since the inference process of a diffusion network involves multiple time steps, the output data of some network layers in earlier time steps is cached during the inference process. This allows at least one subsequent time step to avoid executing the computation of these network layers and directly reuse the cached output data, thereby reducing the computational load and improving inference efficiency. Using the above method, a first data reuse strategy is initialized. Then, during the inference process of the diffusion model, data caching and reuse are performed according to the first data reuse strategy. Finally, the first data reuse strategy is optimized using reinforcement learning methods. This allows for the rapid determination of a suitable data reuse strategy for the diffusion model, improving the efficiency of determining the data reuse strategy for the diffusion model.

[0006] In one possible implementation, the data processing system optimizes a first data reuse strategy based on a reinforcement learning method and the output data of multiple network layers to determine an optimized data reuse strategy. This includes: the data processing system optimizes the first data reuse strategy based on a reinforcement learning method and the output data of multiple network layers under the first data reuse strategy to obtain a second data reuse strategy; during the inference process of the diffusion model based on the second input, the data processing system caches and reuses the output data of multiple network layers according to the second data reuse strategy, and then optimizes the second data reuse strategy based on a reinforcement learning method and the cached output data of multiple network layers under the second data reuse strategy to determine the optimized data reuse strategy.

[0007] The first data reuse strategy is an initial data reuse strategy. The first data reuse strategy is optimized using reinforcement learning. Then, the diffusion model is inferred again based on the optimized second data reuse strategy. If the second data reuse strategy does not meet the requirements, the second data reuse strategy is optimized again using reinforcement learning until a data reuse strategy that meets the requirements is obtained.

[0008] In one possible implementation, the first data reuse strategy includes caching the first output data of the first network layer at a first time step and reusing the first output data at at least one time step. The data processing system caches and reuses the output data of multiple network layers according to the first data reuse strategy, including: caching the first output data of the first network layer during the inference process of the diffusion model at the first time step according to the first data reuse strategy; during the inference process of the diffusion model at the second time step, the data processing system uses the first output data as the output data of the first network layer at the second time step, and then the data processing system determines the input data of the second network layer at the second time step based on the first output data; wherein the second network layer is the next network layer after the first network layer in the diffusion model, and the second time step is one of at least one time step indicating the reuse of the first output data by the first data reuse strategy; the data processing system performs computation of the second network layer at the second time step based on the input data of the second network layer.

[0009] The inference process of the diffusion model is a stepwise denoising process. The diffusion model generates the final output by progressively denoising over multiple time steps. Since the inference process of the diffusion network involves multiple time steps, the output data of some network layers in earlier time steps is cached during the inference process of the diffusion model. This allows at least one subsequent time step to avoid executing the computation of these network layers and directly reuse the cached output data, thereby reducing the computational load in the diffusion model's inference process and improving inference efficiency.

[0010] In one possible implementation, the data processing system caches the first output data, including: at a first time step, the data processing system performs calculations on the first network layer based on the input data of the first network layer to obtain the first output data, and caches the first output data in a first storage area; the data processing system determines the input data of the second network layer at a second time step based on the first output data, including: at a second time step, the data processing system retrieves the first output data from the first storage area, performs a residual connection between the first output data and the input data corresponding to the first network layer at the second time step, and obtains the input data of the second network layer at the second time step.

[0011] In one possible implementation, the above method further includes: after determining that the first output data cached in the first time step is no longer reused, the data processing system deletes the first output data stored in the first storage area so as to use the first storage area to cache data that needs to be cached in other time steps.

[0012] After the data processing system caches the output data of a network layer to a storage area at a time step, if at a certain time step the data processing system determines that the output data will not be reused, the output data will be deleted from the storage area so that the storage area can be used to store output data that needs to be cached later, thereby reducing the storage space occupied when adopting the data reuse strategy.

[0013] In one possible implementation, the data processing system optimizes the first data reuse strategy based on the reinforcement learning method and the output data of multiple network layers under the first data reuse strategy. This includes: the data processing system determines the reward value corresponding to the first data reuse strategy based on the output data of multiple network layers cached under the first data reuse strategy; if the reward value is less than a first threshold, the data processing system optimizes the first data reuse strategy using the reinforcement learning method.

[0014] In one possible implementation, before the data processing system obtains the first data reuse strategy, it further includes: during the process of the diffusion model inferring based on the third input, the data processing system saves the output data of each network layer of the diffusion model at each time step, and then optimizes the third data reuse strategy based on the reinforcement learning method, the third data reuse strategy and the output data of each network layer at each time step to determine the first data reuse strategy.

[0015] In one possible implementation, after the data processing system saves the output data of each network layer at each time step of the diffusion model during inference based on the third input, and before optimizing the third data reuse strategy according to the reinforcement learning method, the third data reuse strategy, and the output data of each network layer at each time step to determine the first data reuse strategy, the system further includes: acquiring the second output data of the third network layer at the third time step and the third output data of the third network layer at the fourth time step; wherein the third time step and the fourth time step are two adjacent time steps in the inference process of the diffusion model; then the data processing system determines the relative error between the second output data and the third output data, and generates the third data reuse strategy based on the relative error; wherein if the relative error is greater than a second threshold, the third data reuse strategy does not include caching the output data of the third network layer at the third time step.

[0016] When the relative error of the output data of the same network layer in adjacent time steps is large, if the data reuse strategy includes caching the output data of the network layer in the previous time step and reusing the data in the next time step, it will cause the reward value of the reward function to be too small. Therefore, by using the above method, we can eliminate the reuse strategy that will cause the reward value to be too small and obtain a data reuse strategy that meets the requirements more quickly.

[0017] In one possible implementation, the above method further includes: the data processing system acquiring user input prompts, executing the inference process of the diffusion model based on the user prompts and the optimized data reuse strategy, and obtaining the output result corresponding to the user prompts.

[0018] Secondly, this application provides a data processing system, which includes an acquisition module, a cache management module, and an optimization module: the acquisition module is used to acquire a first data reuse strategy, which instructs the output data of multiple network layers of the diffusion model to be cached at least one time step during the inference process of the diffusion model, and instructs the diffusion model to reuse the cached output data of the multiple network layers; the cache management module is used to cache and reuse the output data of the multiple network layers according to the first data reuse strategy during the inference process of the diffusion model based on a first input, thereby completing the inference process of the diffusion model; the optimization module is used to optimize the first data reuse strategy according to the reinforcement learning method and the output data of the multiple network layers, and determine the optimized data reuse strategy, which instructs the output data of the network layers to be cached and reused during the inference process of the diffusion model.

[0019] In one possible implementation, the optimization module is further configured to optimize the first data reuse strategy based on the reinforcement learning method and the output data of multiple network layers under the first data reuse strategy to obtain a second data reuse strategy; the cache management module is further configured to cache and reuse the output data of multiple network layers according to the second data reuse strategy during the inference process of the diffusion model based on the second input; the optimization module is further configured to optimize the second data reuse strategy based on the reinforcement learning method and the output data of multiple network layers cached under the second data reuse strategy to determine the optimized data reuse strategy.

[0020] In one possible implementation, the first data reuse strategy includes caching the first output data of the first network layer at a first time step and reusing the first output data at at least one time step. Specifically, the cache management module is used to: cache the first output data of the first network layer during the inference process of the diffusion model at the first time step according to the first data reuse strategy; during the inference process of the diffusion model at the second time step, use the first output data as the output data of the first network layer at the second time step, and then determine the input data of the second network layer at the second time step based on the first output data; wherein the second network layer is the next network layer after the first network layer in the diffusion model, and the second time step is one of at least one time step indicating the reuse of the first output data by the first data reuse strategy; and perform computation of the second network layer based on the input data of the second network layer at the second time step.

[0021] In one possible implementation, the cache management module is specifically used to: perform calculations on the first network layer based on the input data of the first network layer at a first time step to obtain first output data, and cache the first output data in a first storage area; at a second time step, retrieve the first output data from the first storage area, perform a residual connection between the first output data and the input data corresponding to the first network layer at the second time step, and obtain the input data of the second network layer at the second time step.

[0022] In one possible implementation, the cache management module is further configured to: after determining that the first output data cached in the first time step will no longer be reused, delete the first output data stored in the first storage area, so as to use the first storage area to cache data that needs to be cached in other time steps.

[0023] In one possible implementation, the optimization module is specifically used to: determine the reward value corresponding to the first data reuse strategy based on the output data of multiple network layers cached under the first data reuse strategy; if the reward value is less than a first threshold, optimize the first data reuse strategy using a reinforcement learning method.

[0024] In one possible implementation, the cache management module is further configured to save the output data of each network layer of the diffusion model at each time step during the inference process of the diffusion model based on the third input; the optimization module is further configured to optimize the third data reuse strategy based on the reinforcement learning method, the third data reuse strategy and the output data of each network layer at each time step, and determine the first data reuse strategy.

[0025] In one possible implementation, the optimization module is further configured to: firstly acquire the second output data of the third network layer at the third time step and the third output data of the third network layer at the fourth time step; wherein the third time step and the fourth time step are two adjacent time steps in the diffusion model inference process; then determine the relative error between the second output data and the third output data; generate a third data reuse strategy based on the relative error; wherein, if the relative error is greater than a second threshold, the above-mentioned third data reuse strategy does not include the output data of the third network layer in the third time step cache.

[0026] In one possible implementation, the cache management module is further configured to: obtain user input prompts, execute the inference process of the diffusion model based on the user prompts and the optimized data reuse strategy, and obtain the output result corresponding to the user prompts.

[0027] Thirdly, this application provides a computing device including a processor and a memory, wherein the processor executes instructions stored in the memory to implement the method as described in the first aspect or any possible implementation thereof.

[0028] Fourthly, this application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory, wherein the processor of each computing device is used to execute instructions stored in the memory to enable the computing device cluster to implement the method as described in the first aspect or any possible implementation thereof.

[0029] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, cause the computing devices to implement the method described in the first aspect or any possible implementation thereof.

[0030] In a sixth aspect, this application provides a computer program product, which includes a computer program that, when run by a cluster of computing devices, implements the method described in the first aspect or any possible implementation of the first aspect.

[0031] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of a management system provided in this application;

[0033] Figure 2 This is a flowchart illustrating a data reuse strategy for determining a diffusion model, as provided in this application.

[0034] Figure 3 This is a schematic diagram illustrating different granularities of data caching and reuse provided in this application;

[0035] Figure 4 This is a schematic diagram of caching and reuse provided in this application;

[0036] Figure 5 This is a schematic diagram of a data processing system provided in this application;

[0037] Figure 6 This is a schematic diagram of a computing device provided in this application;

[0038] Figure 7 This is a schematic diagram illustrating a network connection between two computing devices provided in this application. Detailed Implementation

[0039] Diffusion models are artificial intelligence (AI) models that generate images or videos based on input prompts. For a trained diffusion model, given an input prompt, it can generate the corresponding image or video. The inference process of a diffusion model is a progressive denoising process. The diffusion model generates the final output by progressively denoising over multiple time steps. For example, a diffusion model consisting of 50 network layers completes one inference cycle in five time steps. Given an input prompt X, in the first time step, the diffusion model processes X through its 50 network layers to obtain output data Y1. In the second time step, Y1 is used as a new input to the diffusion model, which processes Y1 through its 50 network layers to obtain output data Y2. In the third time step, Y2 is input again to the diffusion model, which processes Y2 through its 50 network layers to obtain output data Y3. This process continues until the diffusion model processes the input data for five time steps, completing a full inference process and obtaining the final output. Therefore, during a single inference operation, all network layers in the diffusion model perform multiple computations, resulting in a large computational load and low efficiency.

[0040] Since the inference process of a diffusion network involves multiple time steps, the output data of some network layers in earlier time steps can be cached during the inference process. This allows at least one subsequent time step to skip the computation of those network layers and directly reuse the cached output data, thereby reducing the computational load and improving inference efficiency. Therefore, a data reuse strategy needs to be developed for the diffusion model to indicate which network layer output data should be cached at which time steps and which time steps should reuse the cached data. However, current data reuse strategies are manually set. This method requires users to continuously modify and test different strategies to determine the one that best meets their needs, resulting in low efficiency. Therefore, determining a data reuse strategy for the diffusion model is a pressing technical problem that needs to be solved to improve its inference efficiency.

[0041] To address the aforementioned issues, this application provides a method for determining a data reuse strategy for a diffusion model. For a diffusion model, the method first initializes a first data reuse strategy. This first data reuse strategy indicates which network layer output data to cache at which time steps during the diffusion model's inference process, and which network layer output data cached at previous time steps to reuse at which time steps. Then, during the diffusion model's inference process, data caching and reuse are performed according to the first reuse strategy. Next, based on the cached output data and reinforcement learning, the first data reuse strategy is optimized to obtain an optimized data reuse strategy. The diffusion model is then inferred again based on the optimized data reuse strategy to determine whether the optimized data reuse strategy meets requirements, such as whether it meets the reward requirements of reinforcement learning or the requirements for improved efficiency. If the requirements are met, the optimized data reuse strategy is used as the data reuse strategy for the diffusion model in subsequent inference processes. If the requirements are not met, the data reuse strategy is further optimized, and the diffusion model is inferred again based on the re-optimized data reuse strategy to determine whether the re-optimized data reuse strategy meets the requirements, until the data reuse strategy meets the requirements. Using the methods described above, we can quickly determine the data reuse strategy for the diffusion model based on our needs.

[0042] The following section, with reference to the accompanying drawings, describes the data reuse strategy method for the deterministic diffusion model provided in this application. First, it introduces the management system used to implement the deterministic data reuse strategy method provided in this application. (See attached figures.) Figure 1 , Figure 1 This is a schematic diagram of a management system provided in this application. The management system includes a client 100 and a data processing system 200. The client 100 and the data processing system 200 are connected by a communication link. The client 100 that establishes the communication link with the data processing system 200 can be one or more, and this application does not impose any specific limitations.

[0043] The data processing system 200 can be deployed on a single computing device or in a cluster of computing devices, which can be servers, virtual machines, containers, or edge computing devices. A virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity is used as the virtual machine's hard drive and memory capacity. Each virtual machine has an independent basic input / output system, hard drive, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process. Edge computing devices refer to devices that are closer to the data source and end user, featuring low latency and high bandwidth, such as intelligent routers and edge servers. A computing device cluster can include multiple of the above-mentioned computing devices; this application does not specifically limit this.

[0044] The aforementioned computing devices can be computing devices in cloud data centers, edge computing devices, local servers in enterprise local data centers, or terminal devices used by users, such as computers, laptops, tablets, etc. This application does not make any specific limitations.

[0045] Client 100 is deployed on the user's terminal device to enable human-computer interaction. Client 100 can be software or an application running on the terminal device, such as a client for a personal computer (PC), a browser-based client or browser plugin, an application (APP) running on a mobile terminal, or a console for a cloud platform; this application does not specifically limit its scope. Terminal devices include personal computers, smartphones, wearable devices, handheld processing devices, tablets, mobile laptops, augmented reality (AR) devices, virtual reality (VR) devices, etc., and are not specifically limited here.

[0046] In one possible implementation, client 100 is a cloud platform client provided by a cloud service provider, used to provide users with various cloud services. Users can purchase or rent cloud services through client 100. The method for determining the data reuse strategy of the diffusion model provided in this application can be one of the cloud services. The aforementioned data processing system 200 is deployed on computing equipment in a cloud data center to provide users with cloud services that determine the data reuse strategy of the diffusion model. Users can use cloud services that determine the data reuse strategy of the diffusion model through client 100.

[0047] In another possible implementation, the data processing system 200 is deployed on a server in the enterprise's local data center. The data processing system 200 is a solution provided by the enterprise itself or a third party for determining the data reuse strategy of the diffusion model. The client 100 is deployed on terminal devices used by users within the enterprise, such as desktops and laptops. Users can use the services provided by the data processing system 200 through the client 100 to analyze the diffusion model and determine the data reuse strategy for the diffusion model.

[0048] In another possible implementation, the data processing system 200 and the client 100 can also be deployed on the same computing device, which can be the terminal device used by the user.

[0049] In another possible implementation, the data processing system 200 and the client 100 can also be integrated into a software system and deployed on a computing device for user use, such as on a user's terminal device.

[0050] In one possible implementation, the aforementioned management system further includes a storage system. The storage system stores data received by the data processing system 200, such as user-uploaded diffusion models. The storage system includes storage devices, which can be hard disk drives (HDDs), solid-state drives (SSDs), mechanical hard disks (HDDs), USB flash drives, flash memory, SD cards, etc., without specific limitation in this application. The storage array can be a redundant array of independent disks (RAID), network-attached storage (NAS), storage area network (SAN), etc., without specific limitation in this application.

[0051] It should be understood that the above deployment method is only for example. The client 100 and the data processing system 200 can be flexibly deployed according to actual business needs. This application does not impose any specific limitations.

[0052] The method for determining a data reuse strategy provided in this application is described below with reference to the accompanying drawings. (See also...) Figure 2 , Figure 2 This is a flowchart illustrating a data reuse strategy for determining a diffusion model, as provided in this application. It should be noted that the data reuse strategy mentioned in this application refers specifically to the data reuse strategy corresponding to the diffusion model. The data reuse strategy is used to indicate which network layer output data is cached at which time steps during the diffusion model inference process, and at which time steps the output data cached in previous time steps is reused. Specifically, the data reuse strategy includes whether each network layer performs data caching at each time step, and whether each network layer reuses data cached in previous time steps at each time step; alternatively, the data reuse strategy includes network layer information corresponding to multiple time steps, where each time step's network layer information includes network layers that need data caching and network layers that need data reuse.

[0053] S201. The data processing system obtains the first data reuse strategy corresponding to the diffusion model.

[0054] The data processing system 200 acquires the diffusion model and first initializes a first data reuse strategy for it. This first data reuse strategy can be a randomly generated strategy by the data processing system 200, such as randomly selecting which network layer output data to cache at which time steps, and reusing the output data cached at previous time steps in which network layers at which time steps. Alternatively, the first data reuse strategy can be determined by the data processing system 200 based on the output data of each network layer at each time step during the training phase of the diffusion model. For example, during the training of the diffusion model, after multiple rounds of training, the diffusion model can provide good inference results. At this point, the output data of each network layer at each time step is recorded, and then it is determined which network layers have relatively small errors at which time steps, thereby determining the aforementioned first data reuse strategy. For example, if the relative error of the output data of the fourth network layer is small in two adjacent time steps t1 and t2, then the first data reuse strategy includes caching the output data of the fourth network layer at time step t1 and reusing the output data of the fourth network layer cached at time step t1 at time step t2. The first data reuse strategy can also be a strategy set by the user based on experience. The user inputs the first data reuse strategy in the client 100 and uploads the first data reuse strategy to the data processing system 200.

[0055] For example, a diffusion model includes a noise prediction network, which can be a U-Net network or a diffusion transformer (DiT) network. The noise prediction network typically consists of multiple blocks. If the noise prediction network is a U-Net network, each block includes convolutional layers, activation function layers, and normalization layers. If the noise prediction network is a DiT network, each block includes self-attention layers, cross-attention layers, and a feedforward neural network (FFN). Some layers in the noise prediction network exhibit high similarity in their output features at neighboring time steps; therefore, the first reuse strategy can select some layers in the noise prediction network for data caching and reuse. For example, if the noise prediction network is a DiT network, for the self-attention layer, cross-attention layer, and feedforward neural network layer of the i-th block of the DiT network, at time step t1, the calculation of these three network layers is performed to obtain the output data of these three network layers and cached. Then at time step t2, the calculation of these three network layers is not performed. When the output data of the self-attention layer is needed at time step t2, the output data of the self-attention layer cached at time step t1 is reused. When the output data of the cross-attention layer is needed, the output data of the cross-attention layer cached at time step t1 is reused. When the output data of the feedforward neural network layer is needed, the output data of the feedforward neural network layer cached at time step t1 is reused.

[0056] It should be noted that, for the diffusion model, the noise prediction network includes multiple blocks, and each block includes multiple network layers. In this application, the output data of the network layer can be cached and reused at the granularity of the network layer, or at the granularity of the block, or both the output data of some network layers and the output data of some blocks can be cached and reused. This application does not make any specific limitations on this.

[0057] For example, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating data caching and reuse at different granularities provided in this application. First, it should be noted that the input data of each network layer is a residual concatenation of the input and output data of the previous network layer. For example, for the i-th block in a noise prediction network, at time step t1, if the input of the self-attention layer of the i-th block is X... 11 When outputting data, Y 11 Then the input data X of the cross-attention layer 12 It is X 11 and Y11 The relationship between the input and output data of other network layers, resulting from residual connections, is as described above and will not be repeated here. The following sections explain data caching and reuse at the block level and at the network layer level.

[0058] The data reuse strategy includes caching the output data of the i-th block at time step t1, and caching the output data of the self-attention layer and the cross-attention layer of the (i+1)-th block. For the i-th block, the input of the self-attention layer is X. 11 During the inference process, after the calculations of the three network layers of the i-th block are completed, the output data of the i-th block is X. 14 Cache X 14 Then X 14 As input data for the self-attention layer in the (i+1)th block.

[0059] At time step t1, for the (i+1)th block, the input data of the self-attention layer is X. 14 Perform the calculations corresponding to the self-attention layer to obtain the output data Y of the self-attention layer. 14 And cache Y 14 Then, the input data X for the cross-attention layer is calculated. 15 X 15 It is X 14 and Y 14 The result of the residual connection is then used to calculate the cross-attention layer, resulting in the output data Y of the cross-attention layer. 15 And cache Y 15 This continues until the reasoning process at time step t1 is completed.

[0060] At time step t2, if the input of the self-attention layer of the i-th block is X 21 At this point, instead of performing calculations on each network layer in the i-th block, the output data X of the i-th block buffered at time step t1 is reused. 14 This enables caching and reuse of output data at the block level. For the (i+1)th block, X... 14 As the input data for the self-attention layer in the (i+1)th block, the computation of the self-attention layer is not performed at this time. Instead, the output data Y of the self-attention layer in the (i+1)th block cached at time step t1 is reused. 14 Y 14 As the output data of the self-attention layer in the (i+1)th block of time step t2; then calculate the input data X of the cross-attention layer. 25 X 25 For X14 and Y 14 The result of the residual connection is that the cross-attention layer is not calculated at this point. Instead, the output data Y of the cross-attention layer in the (i+1)th block cached at time step t1 is reused. 15 Y 15 As the output data of the cross-attention layer in the (i+1)th block of time step t2; then calculate the input data X of the feedforward neural network layer. 26 X 26 For X 25 and Y 15 The result of residual connections enables caching and reuse of output data at the network layer level.

[0061] In the data reuse strategy of this application, the network layers that perform data caching at different time steps can be the same or not completely the same, and the network layers that perform data reuse can be the same or not completely the same. This application does not impose specific limitations on this. The following is a specific example to illustrate the first data reuse strategy. For ease of description, the caching and reuse of output data in this first data reuse strategy are performed at the network layer level.

[0062] For example, if the diffusion model includes 9 network layers, numbered 1 to 9, and one inference process includes 4 time steps, namely time step t1, time step t2, time step t3, and time step t4, then the cached and reused data indicated by the first data reuse strategy are shown in Table 1 below.

[0063] The first data reuse strategy is to cache the output data of network layers numbered 2 and 3 at time step t1, reuse the output data of network layers numbered 2 and 3 cached at time step t2, and cache the output data of network layers numbered 4, 7, and 8; reuse the output data of network layer numbered 3 cached at time step t1, reuse the output data of network layers numbered 4, 7, and 8 cached at time step t2, and cache the output data of network layers numbered 2 and 6; reuse the output data of network layer numbered 7 cached at time step t2, and reuse the output data of network layers numbered 2 and 6 cached at time step t4.

[0064] Table 1

[0065]

[0066] It should be noted that the above diffusion model is a pre-trained diffusion model. The diffusion model can be a model already saved in the data processing system 200. The data processing system 200 can process an existing diffusion model. Alternatively, the diffusion model can be uploaded by the user through the client 100. The diffusion model uploaded by the client 100 can be a pre-trained diffusion model or an untrained diffusion model, which is then trained by the data processing system 200 after receiving the diffusion model.

[0067] S202. During the inference process of the diffusion model, the data processing system caches and reuses the output data of multiple network layers according to the first data reuse strategy.

[0068] After acquiring the first data reuse strategy, the data processing system 200 performs diffusion model inference based on the first input. During the inference process, it caches and reuses the output data of multiple network layers according to the aforementioned first data reuse strategy, thereby achieving output data caching and reuse. It should be understood that before performing diffusion model inference based on the first input, the data processing system 200 also needs to generate initial noise, and then performs diffusion model inference based on the first input and the initial noise. This initial noise is generated by the data processing system 200 based on view parameters, which include the resolution, frame rate, etc., of the image or video to be generated.

[0069] During the inference process of the diffusion model, for each network layer at each time step, before executing the computation corresponding to a network layer, the data processing system 200 determines, according to a first data reuse strategy, whether to load the output data cached by the previous network layer in a previous time step when executing the computation of that network layer. After executing the computation of that network layer, it determines, according to the first data reuse strategy, whether to cache the output data of that network layer. If the first data reuse strategy includes caching the first output data of the first network layer at the first time step, the cached first output data is reused in at least one time step after the first time step. Here, the first network layer is any network layer included in the aforementioned diffusion model, and the first time step is any time step in the diffusion model inference process except for the last time step. When the data processing system 200 caches and reuses data from multiple network layers according to the first data reuse strategy, when performing inference at the first time step, when calculating the first network layer, the data processing system 200 determines to cache the first output data of the first network layer according to the first data reuse strategy. After completing the calculation of the first network layer and obtaining the first output data, the data processing system 200 caches the first output data.

[0070] If the first data reuse strategy instructs the reuse of cached first output data in the second time step (i.e., the second time step is one of at least one time step instructing the reuse of first output data), then when the data processing system 200 reaches the second time step for inference, upon computation of the first network layer, if the data processing system 200 determines, according to the first data processing strategy, that the first output data should be reused, then the data processing system 200 does not execute the computation of the first network layer. Instead, it acquires the first output data, uses it as the output data of the first network layer in the second time step, determines the input data of the second network layer based on the first output data, and then executes the computation of the second network layer based on the input data of the second network layer. Here, the second network layer is the next network layer after the first network layer in the diffusion model. The input data of the second network layer is the result of a residual connection between the first output data and the input data of the first network layer in the second time step. Since the computation of the first network layer is not executed in the second time step, the aforementioned input data of the first network layer is not actually input into the first network layer for computation.

[0071] For example, see Figure 4 , Figure 4 This is a schematic diagram of caching and reuse provided in this application. If the diffusion model includes 9 network layers, numbered 1 to 9 respectively, and one inference process includes 4 time steps, namely time step t1, time step t2, time step t3 and time step t4, the first data reuse strategy is shown in Table 1 above.

[0072] If X ij f represents the input data of the network layer numbered j at time step i in the diffusion model. k (x) represents the operation performed by the k-th network layer, and Yi represents the output of the diffusion model at the ith time step. Based on the first data reuse strategy described above, the inference process of the inference model is as follows:

[0073] At time step t1, the input data for network layer numbered 1 in the diffusion model is X. 11 The output data is f1(X) 11 If X is the input data of network layer number 2, then... 12 It is the result of residual connection between the input and output data of network layer number 1, denoted as X. 12 =X 11 +f1(X 11 The output data of network layer number 2 is f2(X). 12 The data processing system 200 caches the output data f2(X) of network layer number 2. 12 The input data X for network layer number 3. 13 =X 12 +f2(X 12The output data of network layer number 3 is f3(X). 13 The data processing system 200 caches the output data f3(X) of network layer number 3. 13 If X is the input data of network layer number 4, then... 14 =X 13 +f3(X 13 ). And so on, the input data X of network layer number 9 19 =X 18 +f8(X 18 The output data of network layer numbered 9 is the output data Y1 = X at time step t1 of the diffusion model. 19 +f9(X 19 ).

[0074] At time step t2, the input data for network layer number 1 is X. 21 That is, Y1 as mentioned above, and the output data is f1(X). 21 If X is the input data of network layer number 2, then... 22 It is the result of residual connection between the input and output data of network layer number 1, denoted as X. 22 =X 21 +f1(X 21 Since time step t1 buffers the output data of network layers numbered 2 and 3, the data processing system 200 does not perform operations on network layers numbered 2 and 3 at this time. The output data of network layer numbered 2 is reused by f2(X). 12 The output data of network layer number 3 is multiplexed using f3(X). 13 If X is the input data of network layer number 3, then... 23 =X 22 +f2(X 12 The input data X of network layer number 4 24 =X 23 +f3(X 13 After the network layer numbered 4 performs the calculation, the output data is f4(X). 24 ), Data processing system 200 cache f4(X) 24 Input data number 5 is X 25 =X 24 +f4(X 24 ). And so on, the input data X of network layer number 7 27 =X 26 +f6(X 26 The output data is f7(X). 27 ), Data processing system 200 cache f7(X) 27 Input data X for network layer number 8. 28 =X27 +f7(X 27 The output data is f8(X). 28 ), Data processing system 200 cache f8(X) 28 The input data X of network layer number 9 29 =X 28 +f8(X 28 The output data of network layer numbered 9 is the output data Y2=X at time step t2 of the diffusion model. 29 +f9(X 29 ).

[0075] At time step t3, the input data for network layer number 1 is X. 31 That is, Y2 as mentioned above, and the output data is f1(X). 31 The input data X of network layer number 2 32 =X 31 +f1(X 31 The output data is f2(X). 32 The data processing system 200 again buffers the output data f2(X) of network layer number 2. 32 The input data X for network layer number 3. 33 =X 32 +f2(X 32 At this point, the operation of network layer number 3 is not performed, and the f3(X) buffer at time step t1 is reused. 13 If X is the input data of network layer number 4, then... 34 =X 33 +f3(X 13 Because time step t2 buffers the output data f4(X) of network layer number 4. 24 At this point, the operation of network layer number 4 is not performed, and the output data of network layer number 4 is multiplexed using f4(X). 24 Then the input X of network layer number 5 35 =X 34 +f4(X 24 And so on, the input X of network layer number 6 36 =X 35 +f5(X 35 The output data is f6(X). 36 ), Data processing system 200 cache f6(X) 36 The input data X for network layer number 7. 37 =X 36 +f6(X 36 At this time, data processing system 200 does not perform the operation of network layer number 7, and reuses the f7(X) buffer of time step t2. 27If the input X of network layer number 8 is... 38 =X 37 +f7(X 27 The data processing system 200 also does not perform the operation of the network layer numbered 8, and reuses the f8(X) buffer of time step t2. 28 If the input X of the network layer numbered 9 is... 39 =X 38 +f8(X 28 The output data of network layer numbered 9 is the output data Y3=X at time step t3 of the diffusion model. 38 +f9(X 39 ).

[0076] At time step t4, the input to network layer numbered 1 is X. 41 That is, Y3 as mentioned above, and the output is f1(X). 41 If X is the input data of network layer number 2, then... 42 It is the result of residual connection between the input and output data of network layer number 1, denoted as X. 42 =X 41 +f1(X 41 Because time step t3 buffers the output data f2(X) of network layer number 2. 32 At this time, the data processing system 200 does not perform the operation of network layer number 2, and the output data of network layer number 2 is multiplexed using f2(X). 32 If X is the input data of network layer number 3, then... 43 =X 42 +f2(X 32 ). And so on, the input data X of network layer number 6 46 =X 45 +f5(X 45 Because time step t3 caches the output data f6(X) of network layer number 6. 36 At this time, the data processing system 200 does not perform the operation of network layer number 6, and the output data of network layer number 6 is multiplexed using f6(X). 36 If the input data X of network layer number 7 is... 47 =X 46 +f6(X 36 And so on, the input X of network layer number 9 49 =X 48 +f8(X 48 The output data of network layer number 9 is the output data Y4=X at time step t4 of the diffusion model. 49 +f9(X 49 ).

[0077] In this application, the data processing system 200 maintains an address mapping table, which includes a mapping relationship between network layer numbers and storage addresses. The address mapping table is used to indicate the cache address of the output data of the network layer. The data processing system 200 stores the output data of the network layer that needs to be cached into the corresponding address according to the address mapping table, and retrieves the cached data from the corresponding storage area according to the address mapping table when the output data needs to be reused.

[0078] In one possible implementation, after the data processing system 200 caches the output data of a network layer to a storage area at one time step, if at some other time step the data processing system 200 determines that the output data will not be reused, it deletes the output data from the storage area, so that the storage area can be used to store output data that needs to be cached later, reducing the storage space occupied when adopting a data reuse strategy. For example, if the data processing system caches the first output data of the first network layer to a first storage area at the first time step, and if the first output data is reused at a second time step and will not be reused at other time steps, the data processing system 200 deletes the first output data from the first storage area. For example, in... Figure 4 In the example shown, the output data f4(X) of network layer number 4 is cached at time step t2. 24 At time step t3, the data f4(X) is reused. 24 If the data is not reused at other time steps, then the data processing system 200 will delete f4(X) after reusing the data at time step t3. 24 ).

[0079] In another possible implementation, the address mapping table includes the memory address corresponding to each network layer at each time step. That is, the memory address corresponding to each network layer at each time step is different, where D... ij This represents the address corresponding to the network layer with time step number j at time step i. If the output data of the network layer with time step number j at time step i needs to be cached, then the output data is cached at address D. ij The corresponding storage area. When the output data of the same network layer is cached at multiple time steps, the output data of that network layer cached later will overwrite the output data cached earlier, thus achieving storage area reuse. For example, the above Figure 4 In the corresponding example, the output data f2(X) of network layer number 2 is cached at time step t1. 12 At time step t3, it is necessary to cache the output data f2(X) of network layer number 2 again. 32 If f2(X) is stored in the region, then f2(X) will be stored in the region. 12 ) Delete, and remove f2(X) 32 )Cache to storage f2(X12 ) storage area.

[0080] S203. The data processing system optimizes the first data reuse strategy based on the output data of multiple network layers cached under the reinforcement learning and first data reuse strategy, and obtains the optimized data reuse strategy.

[0081] After the diffusion model completes inference according to the first data reuse strategy, the data processing system 200 optimizes the first data reuse strategy based on the reinforcement learning (RL) under the first data reuse strategy and the output data of multiple network layers cached under the first data reuse strategy, so as to obtain the optimized data reuse strategy.

[0082] In this application, when optimizing the first data reuse strategy using reinforcement learning, the diffusion model and its inference process constitute the reinforcement learning environment. The state includes environment information after each inference step of the diffusion model. This environment information includes cache information and reuse information corresponding to each time step. The cache information indicates which network layers' output data was cached, and the reuse information indicates which network layers reused the cached output data. An action is a decision made by the agent based on the state of the environment. In this application, an action refers to the data reuse strategy of the diffusion model determined by the agent. The reward is the feedback given by the environment after the agent selects an action; the reward guides the agent to improve the action. The reward is usually a numerical value, and the agent's goal is usually to maximize the reward by selecting actions. In this application, the reward guides the agent to optimize the data reuse strategy. The agent maximizes the reward by optimizing the data reuse strategy of the diffusion model, thereby obtaining a data reuse strategy that meets the requirements.

[0083] In one possible implementation, the relative error between the output data of each network layer when the diffusion model uses a data reuse strategy and the output data of each network layer when the diffusion model does not use a data reuse strategy is used as the reward in reinforcement learning. That is, the reward value of reinforcement learning under a data reuse strategy is determined based on the relative error between the output data of each network layer when the diffusion model uses a data reuse strategy and the output data of each network layer when the diffusion model does not use a data reuse strategy. For any given network layer, the relative error is calculated using the following formula:

[0084] e ij =|d c -d p | / d p (Formula 1) Then the reward function in this application can be as shown in Formula 2.

[0085]

[0086] Among them, e ij It is the relative error corresponding to the network layer numbered j at the i-th time step; d c This refers to the output data of the network layer numbered j at the i-th time step when the diffusion model uses a data reuse strategy; d p is the output data of the network layer numbered j at the i-th time step when the diffusion model does not use a data reuse strategy; n is the number of time steps included in one inference by the diffusion model; k is the number of network layers in the diffusion model; and R1 represents the reward value.

[0087] It should be noted that when determining the data reuse strategy for the diffusion module, at the first input, the data processing system 200 also needs to perform an inference process based on the first input without adopting the first data reuse strategy, to obtain the output data of each network layer at each time step without using the data reuse strategy, and save the output data of each network layer at each time step. For example, in... Figure 4 In the example shown, at time step t2, network layer number 2 reuses the output data f2(X) of that network layer cached at time step t1. 12 If no data reuse strategy is used, then at time step t2, the output data of network layer number 2 is f2(X). 22 Therefore, at time step t2, the relative error e of network layer number 2 is... 22 =|f2(X) 12 )-f2(X 22 )| / f2(X 22 It should be understood that the data processing system 200 first performs the inference process without using a data reuse strategy, saving the output data of each network layer at each time step. Then, using the first data reuse strategy, it performs inference of the diffusion model. In this way, after obtaining the output data of each network layer, the aforementioned relative error can be calculated. After completing the calculation of the relative error, if the cached output data is no longer reused, the data processing system 200 deletes the data to achieve the reuse of the aforementioned storage area.

[0088] For example, the above Figure 4 In the corresponding example, the output data f2(X) of network layer number 2 is cached at time step t1. 12 At time step t2, the output data f2(X) is reused. 12 After that, the output data f2(X) will not be reused in subsequent time steps. 12 Then, based on the output data f2(X) 12 After calculating the corresponding relative error, data processing system 200 deletes f2(X). 12 ).

[0089] In another possible implementation, the KL divergence between the output data of the diffusion model at different time steps when using the first data reuse strategy and the output data of the diffusion model at different time steps when not using the first data reuse strategy is used as the reward in reinforcement learning. The closer the output data of the diffusion model is to the output data when using the first data reuse strategy and the output data when not using the first data reuse strategy, the higher the output accuracy of the diffusion model. Therefore, a KL divergence with a negative reward can be set to maximize the similarity of the diffusion model outputs in the two cases.

[0090] When using KL divergence as the reward in reinforcement learning, firstly, without using the first data reuse strategy, the diffusion model performs one inference operation based on the first input, and the output of the diffusion model at each time step is recorded. The output of the diffusion model at the i-th time step is X. i Then, using the first data reuse strategy, the first input is used to complete one inference of the diffusion model, and the output of the diffusion model at each time step is recorded. The output of the diffusion model at the i-th time step is Y. i The reward function is shown in Formula 3:

[0091]

[0092] Where R² represents the reward value, n is the number of time steps required for the diffusion model to complete one inference, and D... KL (Y i ||X i ) represents Y i With X i The KL divergence between them.

[0093] In another possible implementation, the reward functions corresponding to Formulas 2 and 3 above can be used as the reward functions for reinforcement learning, which can then be used by the agent to optimize the data reuse strategy. Alternatively, the reward values ​​in Formulas 2 or 3 can be weighted to obtain the comprehensive reward value for reinforcement learning, and the agent can optimize the data reuse strategy based on this comprehensive reward value. This application does not impose any specific limitations on this approach.

[0094] If the data processing system 200 determines whether the first data reuse strategy meets the requirements based on the reward function of the reinforcement learning described above, and if the reward value of the reward function under the first data reuse strategy is greater than or equal to the first threshold, then the data processing system 200 determines that the first data reuse strategy meets the requirements and no longer optimizes the first data reuse strategy. If the reward value of the reward function is less than the first threshold, then the data processing system 200 determines that the first data reuse strategy does not meet the requirements and optimizes the first data reuse strategy. After the Agent optimizes the first data reuse strategy to obtain the second data reuse strategy, the second data reuse strategy is used for the inference of the diffusion model. The data processing system 200 performs the inference of the diffusion model again based on the second input and the second data reuse strategy. During the inference process of the diffusion model, the output data of multiple network layers is cached and reused according to the second data reuse strategy to complete the inference process of the diffusion model. The method by which the data processing system 200 caches and reuses the output data of network layers according to the second data reuse strategy during the inference process of the diffusion model is described in S202 and will not be described here. After completing the inference of the diffusion model according to the second data reuse strategy, the data processing system 200 optimizes the second data reuse strategy based on reinforcement learning methods and the output data of multiple network layers cached under the second data reuse strategy. Specifically, the data processing system 200 determines whether the second data reuse strategy meets the requirements based on the reward function of reinforcement learning and the output data of multiple network layers cached under the second data reuse strategy. If the second data reuse strategy meets the requirements, it is adopted as the final data reuse strategy for the diffusion model. If the data processing system 200 determines that the second data reuse strategy does not meet the requirements, it optimizes the second data reuse strategy according to reinforcement learning methods to determine the optimized data reuse strategy, until the optimized data reuse strategy meets the requirements.

[0095] In one possible implementation, if, under a data reuse strategy, the reward value is greater than or equal to a first threshold and the speedup ratio is greater than or equal to a third threshold, then the data processing system 200 determines that the data reuse strategy meets the requirements. If the reward value is greater than or equal to the first threshold, but the speedup ratio is less than the third threshold, then the data processing system 200 determines that the data reuse strategy does not meet the requirements. The data processing system 200 continues to optimize the data reuse strategy according to the above method until the reward value is greater than or equal to the first threshold and the speedup ratio is greater than or equal to the third threshold. The speedup ratio is determined by the following formula 4.

[0096]

[0097] Where T1 is the time it takes for the diffusion model to perform one inference operation without using a data reuse strategy, and T2 is the time it takes for the diffusion model to perform one inference operation with a data reuse strategy.

[0098] It should be noted that after each optimization of the data reuse strategy, the reinforcement learning agent must perform another diffusion model inference. Based on the optimized data reuse strategy, the output data is cached and reused, and it is determined whether the optimized data reuse strategy meets the requirements. The input of the diffusion model can be the same or different each time the diffusion model is inferred. This application does not make specific restrictions on this.

[0099] In one possible implementation, this application also provides another method for determining the aforementioned first data reuse strategy. When determining the first data reuse strategy, a third input is fed into the diffusion model. During the inference process of the diffusion model based on this third input, the data processing system 200 does not use the data reuse strategy and saves the output data of each network layer at each time step during the inference process. In this application, the output data of the network layer numbered j at the i-th time step is denoted as y. ij The output data y of each network layer at each time step is determined. ij Subsequently, the data processing system 200 processes the output data y of each network layer at each time step. ij Using reinforcement learning, determine the primary data reuse strategy.

[0100] Data processing system 200 acquires the output data y of each network layer at each time step during the inference process of the diffusion model based on the third input. ij Then, based on the output data y of each network layer at each time step ij The third data reuse strategy and reinforcement learning method are used to optimize the third data reuse strategy to obtain the first data reuse strategy. Specifically, when determining the first data reuse strategy based on the third data reuse strategy, the third data reuse strategy is a data reuse strategy initialized by the data processing system 200. The method for the data processing system 200 to initialize a data reuse strategy can be referred to the description in S201 above, and will not be explained in detail here.

[0101] Data processing system 200 processes the output data y of each network layer at each time step. ijA reward value is determined by the third data reuse strategy and the corresponding reward function of reinforcement learning. When determining the first data reuse strategy, the reward function of reinforcement learning is based on Formula 5. If the reward value is greater than or equal to the fourth threshold, the data processing system 200 determines that the third data reuse strategy meets the requirements and uses it as the first data reuse strategy. If the reward value is less than the fourth threshold, the data processing system 200 determines that the third data reuse strategy does not meet the requirements. Then, the data processing system 200 optimizes the third data reuse strategy through a reinforcement learning agent to obtain the fourth data reuse strategy. Then, it continues to optimize the third data reuse strategy based on the fourth data reuse strategy and the output data y of each network layer at each time step. ij The reward function determines whether the fourth data reuse strategy meets the requirements until the reward value is greater than or equal to the fourth threshold. The data reuse strategy at this time is then determined as the first data reuse strategy mentioned above.

[0102] When determining the first data reuse strategy, the reward value for reinforcement learning is determined as follows: For a network layer numbered j, the relative error of the network layer at the i-th time step is defined as shown in Equation 5:

[0103] e i ′ j =|y ij -y (i - 1)j | / y ij (Formula 5)

[0104] Among them, e i ′ j y represents the relative error of the network layer numbered j at the i-th time step. ij y represents the output data of the network layer numbered j at the i-th time step. (i - 1)j This represents the output data of the network layer numbered j at the (i-1)th time step. In the inference process of the diffusion model, the (i-1)th time step is the time step preceding the ith time step.

[0105] For a data reuse strategy, which includes instructions on which network layer output data to cache at which time steps and which network layer output data to reuse at which time step, the data processing system 200 determines the reward value under the data reuse strategy by adding the relative errors of the time steps and network layers indicated in the data reuse strategy and then taking the negative value. For example, if a third data reuse strategy indicates that the network layer at the second time step (numbered 5) reuses the cached data between time steps and the network layers at the third time steps (numbered 6 and 7) reuse the data from the previous time steps, then the reward value under this third data reuse strategy is as follows:

[0106]

[0107] After determining the first data reuse strategy of the diffusion model using the above method, the optimized data reuse strategy corresponding to the diffusion model is determined according to the methods described in S201 to S203 above, which is the final data reuse strategy corresponding to the diffusion model.

[0108] When determining the first data reuse strategy using the third data reuse strategy, the diffusion model performs only one inference process for each input. During this inference process, the output data of each network layer at each time step is recorded. In subsequent optimization of the third data reuse strategy, the diffusion model does not perform inference based on the newly generated data reuse strategy. That is, when optimizing the data reuse strategy using reinforcement learning, the reinforcement learning agent does not interact with the diffusion model; the reinforcement learning agent and the diffusion model are decoupled. This avoids the latency caused by multiple inference operations required when the reinforcement learning agent and the diffusion model interact, allowing for a faster determination of a data reuse strategy that meets the lower requirements, i.e., the first data reuse strategy mentioned above. Then, when determining the final data reuse strategy using the first data reuse strategy, the reinforcement learning agent interacts with the diffusion model. Each time the data reuse strategy is updated based on reinforcement learning, the diffusion model needs to perform inference based on the updated data reuse strategy, resulting in an accurate data reuse strategy that meets the requirements.

[0109] In one possible implementation, after the data processing system 200 saves the output data of each network layer at each time step during the inference process of the diffusion model based on the third input, before optimizing the third data reuse strategy according to the reinforcement learning method, the third data reuse strategy, and the output data of each network layer at each time step, and before determining the first data reuse strategy, the data processing system 200 acquires the second output data of the third network layer at the third time step and the third output data of the third network layer at the fourth time step; wherein the third time step and the fourth time step are two adjacent time steps in the inference process of the diffusion model, and the second output data and the third output data are the output data of the network layer without using the data reuse strategy. Then, the data processing system 200 determines the relative error between the second and third output data, and generates a third data reuse strategy based on the relative error. If the relative error between the second and third output data indicates a significant difference between the output data of the third network layer in the fourth time step and the output data of the third network layer in the third time step, and the fourth time step is not suitable for reusing the output data of the third network layer in the third time step, then the data processing system 200 will not include the output data of the third network layer in the third time step cache when generating the third data reuse strategy. Therefore, the fourth time step cannot reuse the output data of the third network layer. Through the above method, the data processing system 200 can also determine other network layers that cannot be reused in adjacent time steps, which will not be described in detail here.

[0110] It should be understood that after determining that the fourth time step of the diffusion model is not suitable for reusing the output data of the third network layer of the third time step, when the data processing system 200 generates the third data reuse strategy, the third data reuse strategy does not include caching the output data of the third network layer in the third time step. During the process of optimizing the third data reuse strategy to obtain the first data reuse strategy, other data reuse strategies generated by the data processing system 200 also do not include caching the output data of the third network layer in the third time step or reusing the output data of the third network layer in the fourth time step, and the first data reuse strategy does not include caching the output data of the third network layer in the third time step. The output data of the network layer and the output data of the third network layer in the third time step are reused in the fourth time step. In addition, in the process of optimizing the first data reuse strategy to obtain the optimized data reuse strategy, the other data reuse strategies generated by the data processing system 200 do not include the output data of the third network layer cached in the third time step and the output data of the third network layer in the third time step reused in the fourth time step. For example, the second data reuse strategy mentioned above does not include the output data of the third network layer cached in the third time step and the output data of the third network layer in the third time step reused in the fourth time step.

[0111] When the relative error of the output data of the same network layer in adjacent time steps is large, if the data reuse strategy includes caching the output data of the network layer in the previous time step and reusing the data in the next time step, it will cause the reward value of the reward function to be too small. Therefore, by using the above method, we can eliminate the reuse strategy that will cause the reward value to be too small and obtain a data reuse strategy that meets the requirements more quickly.

[0112] After determining the optimized data reuse strategy based on the first data reuse strategy, the data processing system 200 uses the optimized data reuse strategy as the final data reuse strategy of the diffusion model. When the user inputs a user prompt and uses the diffusion model to obtain the output result corresponding to the user prompt, the data processing system 200 caches and reuses the data according to the final data reuse strategy, completes the inference process, and obtains the output result corresponding to the user prompt.

[0113] For example, the final data reuse strategy includes caching the output data of network layers numbered 2 and 3 at time step t1, reusing the cached output data of network layers numbered 2 and 3 at time step t2, and reusing the cached output data of network layer numbered 3 at time step t3. Therefore, after receiving user input prompts, during the inference process of the diffusion model, the data processing system 200 caches the output data of network layers numbered 2 and 3 in memory at time step t1. At time step t2, after completing the calculation of network layer number 1, the system retrieves the cached output data of network layers numbered 2 and 3 from memory for reuse. At time step t3, after completing the calculation of network layer number 2, the system retrieves the cached output data of network layer number 3 from memory for reuse.

[0114] For the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application. Therefore, other reasonable combinations of steps that those skilled in the art can conceive of based on the above description also fall within the scope of protection of this application.

[0115] The above text combined Figures 1 to 4 This application details the data reuse strategy for deterministic diffusion models provided in this application. The following section will combine... Figure 5 and Figure 7 The data processing system and computing equipment provided in this application are described separately.

[0116] See Figure 5 , Figure 5This is a schematic diagram of a data processing system 200 provided in this application. The data processing system 200 includes an acquisition module 210, a cache management module 220, and an optimization module 230. The acquisition module 210 acquires a first data reuse strategy, which instructs the diffusion model to cache the output data of multiple network layers at at least one time step during the inference process of the diffusion model, and instructs the diffusion model to reuse the cached output data of the multiple network layers. The cache management module 220 caches and reuses the output data of the aforementioned multiple network layers according to the first data reuse strategy during the inference process of the diffusion model based on a first input, thereby completing the inference process of the diffusion model. The optimization module 230 optimizes the first data reuse strategy based on the reinforcement learning method and the output data of the multiple network layers, determining an optimized data reuse strategy. This optimized data reuse strategy instructs the caching and reuse of the network layer output data during the inference process of the diffusion model.

[0117] In one possible implementation, the optimization module 230 is further configured to optimize the first data reuse strategy based on the reinforcement learning method and the output data of multiple network layers under the first data reuse strategy, to obtain a second data reuse strategy. The cache management module 220 is further configured to cache and reuse the output data of multiple network layers according to the second data reuse strategy during the inference process of the diffusion model based on the second input. The optimization module 230 is further configured to optimize the second data reuse strategy based on the reinforcement learning method and the output data of multiple network layers cached under the second data reuse strategy, to determine the optimized data reuse strategy.

[0118] In one possible implementation, the first data reuse strategy includes caching the first output data of the first network layer at a first time step and reusing the first output data at at least one time step; the cache management module 220 caches the first output data of the first network layer during the inference process of the diffusion model at the first time step according to the first data reuse strategy; during the inference process of the diffusion model at the second time step, the first output data is used as the output data of the first network layer at the second time step, and then the input data of the second network layer at the second time step is determined based on the first output data; wherein, the second network layer is the next network layer after the first network layer in the diffusion model, and the second time step is one of at least one time step indicating the reuse of the first output data by the first data reuse strategy; and the calculation of the second network layer is performed in the second time step based on the input data of the second network layer.

[0119] In one possible implementation, the cache management module 220 performs calculations on the first network layer based on the input data of the first network layer in the first time step to obtain the first output data and caches the first output data in the first storage area; in the second time step, it retrieves the first output data from the first storage area, performs a residual connection between the first output data and the input data corresponding to the first network layer in the second time step, and obtains the input data of the second network layer in the second time step.

[0120] In one possible implementation, the cache management module 220 is further configured to delete the first output data stored in the first storage area after determining that the first output data cached in the first time step will no longer be reused, so as to use the first storage area to cache data that needs to be cached in other time steps.

[0121] In one possible implementation, the optimization module 230 determines the reward value corresponding to the first data reuse strategy based on the output data of multiple network layers cached under the first data reuse strategy. If the reward value is less than the first threshold, the first data reuse strategy is optimized by reinforcement learning.

[0122] In one possible implementation, the cache management module 220 is further configured to save the output data of each network layer at each time step of the diffusion model during the inference process based on the third input. The optimization module 230 is further configured to optimize the third data reuse strategy based on the reinforcement learning method, the third data reuse strategy, and the output data of each network layer at each time step, to determine the first data reuse strategy.

[0123] In one possible implementation, the optimization module 230 first obtains the second output data of the third network layer at the third time step and the third output data of the third network layer at the fourth time step; wherein the third time step and the fourth time step are two adjacent time steps in the diffusion model inference process; then it determines the relative error between the second output data and the third output data; and generates a third data reuse strategy based on the relative error; wherein, if the relative error is greater than a second threshold, the above-mentioned third data reuse strategy does not include the output data of the third network layer in the third time step cache.

[0124] In one possible implementation, the cache management module 220 is also used to obtain user prompts input by the user, execute the inference process of the diffusion model according to the user prompts and the above-mentioned optimized data reuse strategy, and obtain the output results corresponding to the user prompts.

[0125] The specific method for implementing the data reuse strategy of the deterministic diffusion model in the data processing system 200 can be referred to the above. Figures 1-4 The relevant descriptions in the corresponding method embodiments are not repeated here. It should be understood that... Figure 5This application merely illustrates one way of dividing a data processing system. In practical applications, data processing systems can be divided in other ways, and this application does not impose any specific limitations.

[0126] This application also provides a computing device, such as Figure 6 As shown, Figure 6 This is a schematic diagram of a computing device provided in this application. The computing device 600 includes a bus 602, a processor 604, a memory 606, and a communication interface 608. The processor 604, the memory 606, and the communication interface 608 communicate with each other via the bus 602. It should be understood that this application does not limit the number of processors 604 in the computing device 600. The aforementioned processor 604 may be a central processing unit (CPU), or may include a CPU and other hardware chips. The aforementioned hardware chips may be of various types, such as any one or more of the following: graphics processing unit (GPU), neural network processing unit (NPU), tensor processing unit (TPU), programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), or digital signal processor (DSP). This application does not specifically limit the types of chips mentioned above.

[0127] Bus 602 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 602 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 602 may include a path for transmitting information between various components of the computing device 600 (e.g., memory 606, processor 604, communication interface 608).

[0128] Memory 606 may include volatile memory, such as random access memory (RAM). Memory 606 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0129] The memory 606 stores executable program code, and the processor 604 executes the executable program code to implement... Figures 1 to 4 The method described in the corresponding method embodiment. That is to say, the memory 606 stores program code for the functions of the above-mentioned data processing system 200, thereby realizing... Figures 1 to 4 The corresponding method embodiment is the method implemented by the data processing system 200. The program code includes one or more software modules, and the one or more software modules include... Figure 5 The acquisition module 210, cache management module 220, and optimization module 230 shown are implemented by the processor 604 executing executable program code. Figures 1-4 The corresponding process for determining the data reuse strategy will not be elaborated here.

[0130] The communication interface 608 can be a wired or wireless interface, used for communication with other modules or devices. For example, it can handle user-uploaded diffusion models. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., while the wireless interface can be a cellular network interface or a wireless LAN interface.

[0131] It should be understood that the computing device 600 provided according to this application may correspond to this application. Figure 1 The provided data processing system 200 includes various modules, and can be used to execute... Figures 1 to 4 The corresponding entities in the provided method, and the above and other operations and / or functions of each module in the computing device 600, are respectively implemented to achieve Figures 1 to 4 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0132] This application also provides a computing device cluster. The computing device cluster includes at least one such... Figure 6 The computing devices shown. The memory 606 of one or more computing devices 600 in the computing device cluster may store the same memory for performing the above-described tasks. Figures 1 to 4 The instructions implemented by the data processing system 200 in the corresponding embodiment.

[0133] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing a method to determine a data reuse strategy. In other words, a combination of one or more computing devices 600 can jointly execute instructions for implementing a case management method.

[0134] It should be noted that the memory 606 in different computing devices 600 within the computing device cluster can store different instructions, each used to implement a portion of the functions of the case management system. That is, the instructions stored in the memory 606 of different computing devices 600 can implement the functions of one or more modules among the acquisition module 210, cache management module 220, and optimization module 230.

[0135] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 7 This is a schematic diagram illustrating a network connection between two computing devices provided in this application. Figure 7 One possible implementation is shown. For example... Figure 7 As shown, two computing devices 600A and 600B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 606 in computing device 600A stores instructions for executing the functions of the fetch module 210 and the cache management module 220. Meanwhile, the memory 606 in computing device 600B stores instructions for executing the functions of the optimization module 230.

[0136] It should be understood that Figure 7 The functions of computing device 600A shown can also be performed by multiple computing devices 600. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600.

[0137] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on a computing device, it causes the computing device to perform… Figures 1 to 4 The method described in the corresponding method embodiment.

[0138] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to implement… Figures 1 to 4 The methods described in the corresponding method embodiments. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method of determining a data multiplexing strategy for a diffusion model, characterized by, The method includes: A first data reuse strategy is obtained, wherein the first data reuse strategy is used to indicate that the output data of multiple network layers of the diffusion model is cached at at least one time step during the inference process of the diffusion model, and to indicate that the diffusion model reuses the cached output data of the multiple network layers. During the reasoning process of the diffusion model based on the first input, the output data of the multiple network layers are cached and reused according to the first data reuse strategy to complete the reasoning process of the diffusion model. Based on the reinforcement learning method and the output data of the multiple network layers, the first data reuse strategy is optimized to determine the optimized data reuse strategy; the optimized data reuse strategy is used to indicate that the output data of the network layers is cached and reused during the inference process of the diffusion model.

2. The method of claim 1, wherein, The step of optimizing the first data reuse strategy based on the reinforcement learning method and the output data of the multiple network layers, and determining the optimized data reuse strategy, includes: Based on the reinforcement learning method and the output data of the multiple network layers, the first data reuse strategy is optimized to obtain the second data reuse strategy; During the reasoning process of the diffusion model based on the second input, the output data of multiple network layers are cached and reused according to the second data reuse strategy. Based on the output data of multiple network layers cached under the second data reuse strategy and the reinforcement learning method, the second data reuse strategy is optimized to determine the optimized data reuse strategy.

3. The method according to claim 1 or 2, characterized in that, The first data reuse strategy includes caching the first output data of the first network layer at a first time step, and reusing the first output data at at least one time step; The step of caching and reusing the output data of the multiple network layers according to the first data reuse strategy includes: During the inference process of the diffusion model at the first time step according to the first data reuse strategy, the first output data is cached; The first output data is used as the output data of the first network layer in the second time step, and the input data of the second network layer in the second time step is determined based on the first output data; the second network layer is the next network layer after the first network layer in the diffusion model, and the second time step is one of at least one time step in which the first data reuse strategy indicates the reuse of the first output data. At the second time step, the calculation of the second network layer is performed based on the input data of the second network layer.

4. The method of claim 3, wherein, The cached first output data includes: At the first time step, the calculation of the first network layer is performed based on the input data of the first network layer to obtain the first output data; Cache the first output data in the first storage area; The step of determining the input data of the second network layer at the second time step based on the first output data includes: In the second time step, the first output data is obtained from the first storage area, and the first output data and the input data corresponding to the first network layer in the second time step are residually connected to obtain the input data of the second network layer in the second time step.

5. The method of claim 4, wherein, The method further includes: After determining that the first output data cached in the first time step will no longer be reused, the first output data stored in the first storage area is deleted so that the first storage area can be used to cache data that needs to be cached in other time steps.

6. The method according to any one of claims 1 to 5, characterized in that, The step of optimizing the first data reuse strategy based on the reinforcement learning method and the output data of the multiple network layers includes: The reward value corresponding to the first data reuse strategy is determined based on the output data of multiple network layers cached under the first data reuse strategy. If the reward value is less than the first threshold, the first data reuse strategy is optimized using the reinforcement learning method.

7. The method according to any one of claims 1 to 6, characterized in that, Before obtaining the first data reuse strategy, the method further includes: During the reasoning process of the diffusion model based on the third input, the output data of each network layer of the diffusion model at each time step is saved; Based on the reinforcement learning method, the third data reuse strategy, and the output data of each network layer at each time step, the third data reuse strategy is optimized to determine the first data reuse strategy.

8. The method of claim 7, wherein, Based on the reinforcement learning method, the third data reuse strategy, and the output data of each network layer at each time step, the third data reuse strategy is optimized. Before determining the first data reuse strategy, the process further includes: Obtain the second output data of the third network layer at the third time step and the third output data of the third network layer at the fourth time step; wherein the third time step and the fourth time step are two adjacent time steps in the diffusion model inference process; Determine the relative error between the second output data and the third output data; The third data reuse strategy is generated based on the relative error; wherein, when the relative error is greater than the second threshold, the third data reuse strategy does not include caching the output data of the third network layer at the third time step.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Obtain user input prompts, execute the inference process of the diffusion model based on the user prompts and the optimized data reuse strategy, and obtain the output result corresponding to the user prompts.

10. A data processing system, characterized by include: The acquisition module is used to acquire a first data reuse strategy, which is used to instruct that the output data of multiple network layers of the diffusion model be cached at at least one time step during the inference process of the diffusion model, and to instruct the diffusion model to reuse the cached output data of the multiple network layers. The cache management module is used to cache and reuse the output data of the multiple network layers according to the first data reuse strategy during the inference process of the diffusion model based on the first input, so as to complete the inference process of the diffusion model. An optimization module is used to optimize the first data reuse strategy based on the reinforcement learning method and the output data of the multiple network layers, and to determine the optimized data reuse strategy. The optimized data reuse strategy is used to instruct the output data of the network layer to be cached and reused during the inference process of the diffusion model.

11. The system according to claim 10, characterized in that, The optimization module is further configured to optimize the first data reuse strategy based on the reinforcement learning method and the output data of the multiple network layers to obtain a second data reuse strategy. The cache management module is also used to cache and reuse the output data of multiple network layers according to the second data reuse strategy during the inference process of the diffusion model based on the second input. The optimization module is further configured to optimize the second data reuse strategy based on the output data of multiple network layers cached under the second data reuse strategy and the reinforcement learning method, and determine the optimized data reuse strategy.

12. The system of claim 10 or 11, wherein, The first data reuse strategy includes caching the first output data of the first network layer at a first time step, and reusing the first output data at at least one time step; The cache management module is specifically used for: During the inference process of the diffusion model at the first time step according to the first data reuse strategy, the first output data is cached; The first output data is used as the output data of the first network layer in the second time step, and the input data of the second network layer in the second time step is determined based on the first output data; the second network layer is the next network layer after the first network layer in the diffusion model, and the second time step is one of at least one time step in which the first data reuse strategy indicates the reuse of the first output data. At the second time step, the calculation of the second network layer is performed based on the input data of the second network layer.

13. The system of claim 12, wherein, The cache management module is specifically used for: At the first time step, the calculation of the first network layer is performed based on the input data of the first network layer to obtain the first output data; Cache the first output data in the first storage area; In the second time step, the first output data is obtained from the first storage area, and the first output data and the input data corresponding to the first network layer in the second time step are residually connected to obtain the input data of the second network layer in the second time step.

14. The system of claim 13, wherein, The cache management module is also used for: After determining that the first output data cached in the first time step will no longer be reused, the first output data stored in the first storage area is deleted so that the first storage area can be used to cache data that needs to be cached in other time steps.

15. The system according to any one of claims 10-14, characterized in that, The optimization module is specifically used for: The reward value corresponding to the first data reuse strategy is determined based on the output data of multiple network layers cached under the first data reuse strategy. If the reward value is less than the first threshold, the first data reuse strategy is optimized using the reinforcement learning method.

16. The system according to any one of claims 10-15, characterized in that, The cache management module is also used to save the output data of each network layer of the diffusion model at each time step during the process of the diffusion model inferring based on the third input. The optimization module is further configured to optimize the third data reuse strategy based on the reinforcement learning method, the third data reuse strategy, and the output data of each network layer at each time step, and determine the first data reuse strategy.

17. The system of claim 16, wherein, The optimization module is also used for: Obtain the second output data of the third network layer at the third time step and the third output data of the third network layer at the fourth time step; wherein the third time step and the fourth time step are two adjacent time steps in the diffusion model inference process; Determine the relative error between the second output data and the third output data; The third data reuse strategy is generated based on the relative error; wherein, when the relative error is greater than the second threshold, the third data reuse strategy does not include caching the output data of the third network layer at the third time step.

18. The system of any of claims 10-17, wherein, The cache management module is also used for: Obtain user input prompts, execute the inference process of the diffusion model based on the user prompts and the optimized data reuse strategy, and obtain the output result corresponding to the user prompts.

19. A computing device, comprising: The computing device includes a processor and a memory, the memory being used to store instructions, and the processor being used to execute the instructions stored in the memory to implement the method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, It includes computer program instructions that, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 9.

Images