Diffusion model reasoning acceleration system and method based on model learning

Abstract

The invention discloses a diffusion model reasoning acceleration system and method based on model learning, and belongs to the field of artificial intelligence content generation, and the system comprises a step number cutting module which is used for compressing calculation steps of a denoising calculation process in diffusion model reasoning; and the task node matching module is used for distributing and scheduling the computing tasks to computing nodes of the cloud-side collaborative network system. According to the diffusion model reasoning acceleration system and method based on model learning, a cloud edge collaboration framework is provided, two strategies of step number compression and task scheduling are utilized, calculation acceleration under the condition that a user uses the diffusion model to execute a text graph reasoning calculation task is achieved, and the image quality can be guaranteed in the reasoning process.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence content generation technology, and in particular to a diffusion model reasoning acceleration system and method based on model learning. Background Art

[0002] The deployment of large-scale deep learning models has enabled the widespread use of artificial intelligence generated content (AIGC). AIGC brings new challenges to computing infrastructure and application services. The iteration, training, and reasoning of large-scale AIGC models are usually performed in cloud computing centers with high computing power. However, traditional AI cloud computing centers are no longer sufficient to meet the computing power requirements of large models; in addition, when performing interaction-intensive reasoning tasks, the remoteness of cloud computing centers will cause large-scale transmission delays, seriously reducing the service quality and user experience of AIGC services. A potential solution to these challenges is to adopt a cloud-edge collaborative network resource architecture to jointly deploy reasoning services with low computing power requirements and high user interaction frequency, improve the instantaneity of AIGC services, and reduce the dependence of AIGC service providers on cloud computing centers. Summary of the invention

[0003] The purpose of the present invention is to provide a diffusion model reasoning acceleration system and method based on model learning to solve the problems mentioned in the above background technology.

[0004] To achieve the above object, the present invention provides a diffusion model reasoning acceleration system based on model learning, comprising:

[0005] The step number clipping module is used to compress the calculation steps of the denoising calculation process in the diffusion model inference;

[0006] The task node matching module is used to allocate and schedule computing tasks to the computing nodes of the cloud-edge collaborative network system.

[0007] Preferably, the diffusion model reasoning includes:

[0008] The encoding process maps the prompt word and seed provided by the user to the feature vector of the latent space. The semantic feature vector corresponding to the prompt word is represented by p, and the seed feature vector is represented by s. p and s are used together as the input of the denoising process.

[0009] The denoising calculation will be iterated n times, and after each step i, an intermediate vector s with the same dimension as the seed feature vector is output i , and use the intermediate vector and p as the input vector for the next step of denoising calculation;

[0010] The decoding process is to convert the intermediate vector s output from the last step of the denoising calculation process n Mapping from latent vector space to image space is provided to users.

[0011] A diffusion model reasoning acceleration method based on model learning includes the following steps:

[0012] S1, according to the step number clipping module, when the denoising calculation unit in the diffusion model reasoning performs calculation, by formulating a step compression strategy and building a step number compression prediction model, the denoising calculation steps are compressed;

[0013] S2. Classify the computing tasks through the step compression model, determine the execution computing amount and input of the task, and then allocate and schedule the computing tasks to the computing nodes of the cloud-edge collaborative network system according to the task node matching module.

[0014] Preferably, the specific steps of S1 are as follows:

[0015] S11, formulate a step compression strategy, by saving the intermediate vectors of user requests in the reasoning process, when a new user request arrives, if there is a prompt word in the historical user request whose similarity with the prompt word of the new user request reaches a preset threshold, the denoising calculation of the current user request is started from the j+1th step, and the jth intermediate vector of the historical user request that meets the requirements and the feature vector of the current user prompt word are selected as the input of the denoising calculation, and the denoising calculation of the current user request is compressed from n steps to nj steps;

[0016] S12. Build a step compression prediction model based on the multi-head attention mechanism. Before model training, obtain the number of inference steps n, the feature vector s of a specific seed, and the feature vector p corresponding to two prompt words whose similarity exceeds the threshold k. 1 , p 2 , use cosine similarity cos() to measure p 1 and p 2 The similarity between them, the selected prompt word pair satisfies the condition cos(p 1 ,p 2 )>k; for p that satisfies the condition 1 With p 2 , firstly, p 1 and p 2 Perform inference and save p separately 1 The intermediate vector group S = {s 1 ,s 2 ,…,s n}, then use n, p 2 Each vector in the intermediate vector group performs {n-1, n-2, ..., 1} steps of denoising calculation respectively, decodes the image, and then performs denoising on p 2 The original image and the image output by the compression step are scored using the ImageReward scoring framework, and the scores are organized into a vector I; during the model training process, input s, p1 , p 2 With I, the training model is passed through s,p 1 , p 2 Information prediction I, using Smooth L1 loss to guide the update of model parameters.

[0017] Preferably, the specific steps of S2 are as follows:

[0018] S21. Model the task scheduling environment, including state space, JCDRL, action space and environment;

[0019] S22. Formulate a task scheduling strategy to obtain the total computing time T of computing task j. over and the optimization objective expressions under several similar tasks;

[0020] S23. Use a policy network P and a value network V to model a task scheduling model for matching computing tasks with computing nodes.

[0021] Preferably, the state space in S21 is defined by the states of the computing nodes N and the surviving tasks J in the system, and the surviving tasks include tasks to be assigned to computing nodes and tasks being executed;

[0022] The action space A(t) indicates that each task entering the system in the interval [t, t+1] is sequentially scheduled to the computing nodes and updates the status of the surviving tasks and computing nodes in the entire system;

[0023] In the cloud-edge network system, an intelligent agent is used to perform task scheduling actions, which triggers the state transition of the entire cloud-edge network system. The transition function is:

[0024] Pr[ST(t+1)=st'|ST(t)=st,A(t)]

[0025] ST(t) represents the current execution status of the task being executed; t represents the time; ST(t+1) represents the updated state of the state space after the agent completes all actions at time t; st represents the current state, and st' represents the new state output after the system state is updated.

[0026] Preferably, the specific content of S22 is:

[0027] In the task scheduling strategy, the task j to be executed is arrive When the task scheduling side is reached at the time, when the computational amount of the task has been determined, the expected total computational time T of the computation j is calculated. over , including the transmission delay T of task information between nodes trans , the queuing delay T of the task on the assigned computing node wait and execution time Texe ;

[0028] For the transmission delay, if j is not compressed in the denoising calculation process, the transmission delay of the information corresponding to j is selected. Otherwise, the transmission delay of the intermediate vector of the historical task corresponding to j used as the input of the compressed denoising calculation process is also selected. When the amount of transmitted data is M, the transmission delay between two different computing nodes N and N' is determined by r(N) and r(N'):

[0029]

[0030] When task j matches the corresponding computing node N, obtain the resource status of the computing node. If the remaining resources of the current computing node can execute the task, there is no queuing delay. If the remaining resources of the current computing node are not enough to support the execution of task j, obtain all tasks on the computing node and arrange them in descending order according to the expected completion time. The queuing delay of task j is the earliest release. After the release, the remaining resources of the computing node support the completion time t of task j' of the current computing task j. end The difference between the current time t and the time t at which task j starts execution is determined by transmission delay and queuing delay. start , queuing delay and the time t at which task j starts execution start The expression is as follows:

[0031]

[0032] t start (j) = t arrive (j)+T wait (j,N)

[0033] The computational amount of task j is proportional to the corresponding number of denoising steps s, denoted by C; the computational rate r of all tasks of the load on the computing node in each time interval [t, t+1] is constant. The relationship between the computational rates of multiple computing tasks in the time interval on a single computing node is expressed as:

[0034]

[0035] The execution time T of task j exe , start execution time t start and the execution end time t end The relationship between them is expressed as:

[0036] t end =t start +T exe (j,N)

[0037]

[0038] In summary, the total computation time of task j is T over The expression is as follows:

[0039] T over (j,N)=T wait (j,N)+T end (j,N)

[0040] The optimization objective expression for multiple tasks of the same type is as follows:

[0041]

[0042] Preferably, the specific content of S23 is:

[0043] The policy network outputs the probability transfer distribution d of the action at the current state st, and executes multiple consecutive actions based on d, corresponding to scheduling the computing task to be executed at the current moment to the computing node, and then updates the system state after each action, outputting a new state st'; for each task j, after the completion of the task, the agent obtains the execution reward R(j) of the task, and encapsulates the starting state, action, post-execution state and execution reward during the execution of the task into experience q(j) = {st(j), A(j), st'(j), R(j)} and puts it into the experience pool Q; V uses the value function to evaluate the value of the action according to the action output by the policy network, and updates the action strategy.

[0044] Therefore, the present invention adopts the above-mentioned diffusion model reasoning acceleration system and method based on model learning, which has the following beneficial effects:

[0045] (1) Using cloud-edge collaborative systems to process service requests, thereby reducing reliance on expensive servers;

[0046] (2) Use distributed computing systems to selectively schedule tasks to enhance parallelism and optimize efficiency and convenience, thereby minimizing delays in resource transmission and loading and improving service quality;

[0047] (3) A diffusion model inference acceleration system guided by the step clipping principle is introduced, which includes a step clipping strategy based on a multi-head attention mechanism; considering the available computing resources, a task node matching strategy is designed using deep reinforcement learning.

[0048] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic diagram of the overall system architecture of an embodiment of the present invention;

[0050] Figure 2A schematic diagram of the step compression principle of an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the structure of a step compression prediction model according to an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of a task scheduling environment model according to an embodiment of the present invention;

[0053] Figure 5 1 is a diagram showing the training process results of an embodiment of the present invention, wherein (a) shows the trend of the loss function results during the training process; (b) shows a schematic diagram showing a comparison between the average score of the compressed output image judged by ImageReward and the average score of the output image under the original denoising step number;

[0054] Figure 6 A schematic diagram of the results of strategy loss, value loss, entropy loss and overall loss in an embodiment of the present invention;

[0055] Figure 7 1 shows the reward situation during training under different inference requests and different numbers of occupied nodes in an embodiment of the present invention; wherein, (a) represents the situation where 1 task arrives in 1 second and 10% of the cloud computing nodes are occupied; (b) represents the situation where 1 task arrives in 1 second and 20% of the cloud computing nodes are occupied; (c) represents the situation where 2 tasks arrive in 1 second and 10% of the cloud computing nodes are occupied; (d) represents the situation where 2 tasks arrive in 1 second and 20% of the cloud computing nodes are occupied. DETAILED DESCRIPTION

[0056] Example

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Reference Figure 1 The present invention discloses a diffusion model reasoning acceleration system based on model learning, comprising:

[0059] The step number clipping module is used to compress the calculation steps of the denoising calculation process in the diffusion model inference;

[0060] The task node matching module is used to allocate and schedule computing tasks to the computing nodes of the cloud-edge collaborative network system.

[0061] For the above system, a diffusion model reasoning acceleration method based on model learning is also provided, comprising the following steps:

[0062] S1. According to the step number trimming module, when the denoising calculation unit in the diffusion model inference performs calculation, the denoising calculation steps are compressed by formulating a step compression strategy and building a step number compression prediction model.

[0063] S11. Formulate a step compression strategy:

[0064] When a user uses the Wensheng graph reasoning service, he submits a reasoning request to the server. The request information contains the prompt word, the seed, and the number of calculation steps n. The server generates the image required by the user by performing forward reasoning of the diffusion model. Diffusion model reasoning includes three calculation units: encoding, decoding, and denoising.

[0065] The encoding process maps the prompt word and seed provided by the user to the feature vector of the latent space. The semantic feature vector corresponding to the prompt word is represented by p, and the seed feature vector is represented by s. p and s are used as the input of the denoising process together. The denoising calculation will be iterated n times. After each step (represented by i), the intermediate vector s with the same dimension as the seed feature vector is output. i , and use the intermediate vector and p as the input vector for the next step of denoising calculation; the decoding process is to convert the intermediate vector s output from the last step of the denoising process n Mapping from latent vector space to image space is provided to users.

[0066] The reasoning process shows that the content of the final output image is jointly determined by the selection of seeds and the semantics of the prompt words. Experiments have shown that when the same seed is selected and the number of denoising calculation iterations is the same, prompt words with similar semantics will output similar images. In addition, the intermediate vectors in the denoising process can be obtained. On this basis, this embodiment saves some intermediate vectors of user requests in the reasoning process. When a new user request arrives, if the prompt words in the historical user requests are similar to the prompt words of the new user request and reach a preset threshold, the denoising calculation of the current user request starts from the j+1th step, and directly selects the jth intermediate vector of the historical user request that meets the requirements and the feature vector of the current user prompt word as the input of the denoising calculation. Under such a strategy, the denoising calculation of the current user request is compressed from n steps to nj steps, thereby reducing the amount of calculation in the denoising process and achieving acceleration in the reasoning process. Figure 2 shown.

[0067] S12. Constructing step number compression prediction model:

[0068] By designing a step number compression prediction model, we can determine whether the denoising calculation requested by the current user can be compressed. If it can be compressed, how many steps will be compressed. The design principle of this model is to ensure the output image quality after compressing the denoising calculation steps, and to achieve the maximum compression steps when the image quality meets the requirements. The overall design is as follows: Figure 3 shown.

[0069] The step compression prediction model is a module based on the multi-head attention mechanism, which is trained by supervised learning. Before training, it is necessary to obtain the inference step number n, the feature vector s of a specific seed, and the feature vector p corresponding to two prompt words whose similarity exceeds the threshold k. 1 , p 2 In order to select a suitable prompt word pair, this embodiment uses cosine similarity cos() to measure p 1 and p 2 The similarity between them, the selected prompt word pair needs to meet the condition cos(p 1 ,p 2 )>k. For p that satisfies the condition 1 With p 2 , firstly, p 1 and p 2 Perform inference and save p separately 1 The intermediate vector group S = {s 1 ,s 2 ,…,s n}, then use n, p 2 Each vector in the intermediate vector group performs {n-1, n-2, ..., 1} steps of denoising calculation respectively, decodes the image, and then performs denoising on p 2 The original image and the image output by the compression step are scored using the ImageReward scoring framework, and the scores are organized into a vector I. The above process will be performed in batches. During the training model, input s, p 1 , p 2 This embodiment requires training the model through s, p 1 , p 2 Information prediction I, using Smooth L1 loss to guide the update of model parameters.

[0070] The model uses the output predicted image score as a basis to classify whether the image is compressed and the denoising calculation steps are determined, and the compression amount of the user reasoning task that can compress the calculation steps is determined. When a user requests to enter the system, it first determines whether there is a historical task in the system that is consistent with s and n and meets the prompt word similarity condition. If not, the steps of the task cannot be compressed. After the reasoning is completed, the intermediate vector of the task is stored on the computing node used to execute the task. If it exists, take the historical task with the highest prompt word similarity, use the step compression model to predict the final score under each intermediate vector of the historical task as the starting input, and take the corresponding step with the highest prediction score as the starting calculation step. The calculation task that has undergone step compression does not retain the intermediate vector.

[0071] S2. Classify the computing tasks through the step compression model, determine the execution computing amount and input of the task, and then allocate and schedule the computing tasks to the computing nodes of the cloud-edge collaborative network system according to the task node matching module.

[0072] After the above computing tasks are classified through the step compression model and the execution computational amount and input of the tasks are determined, the computing tasks need to be assigned and scheduled to the computing nodes of the cloud-edge collaborative network system for execution. The computing tasks enter the network system through the edge computing nodes and may be subsequently scheduled to cloud nodes or other edge nodes, or they may be executed locally. Due to the heterogeneity of computing nodes and the diversity of computing tasks' resource requirements, the matching of computing tasks and computing nodes is modeled through a deep reinforcement learning process, and matching patterns under different computing node and task characteristics are learned in the iterative process.

[0073] S21. Modeling the task scheduling environment:

[0074] The overall modeling structure is as follows Figure 4 As shown. The Markov decision process is used to characterize the task scheduling and execution within the system. In the cloud-edge network system, an agent is used to perform task scheduling actions, which triggers the state transition of the entire network system. The transition function is Pr[ST(t+1)=st'|ST(t)=st,A(t)]. The optimization goal of the agent during task execution is to minimize the average completion time of all tasks under the same model parameters, while ensuring that the quality of the output image will not be significantly reduced while achieving this goal.

[0075] The state space is defined by the states of the computing nodes N and the surviving tasks J in the system. The surviving tasks include tasks to be assigned to computing nodes and tasks being executed. ST(t) represents the current execution state of the task being executed, the resource requirements of the tasks to be scheduled, and the current resource state of the computing nodes. At each moment t, this embodiment obtains information about the state space to perform task scheduling actions. The current states of the computing nodes and tasks depend on the decision results of the previous moment. The action space A(t) indicates that each task entering the system in the interval [t, t+1] is sequentially scheduled to the computing nodes and updates the states of the surviving tasks and computing nodes in the entire system. After the agent completes all actions at moment t, the state space is updated to ST(t+1), which becomes the state at the next decision moment.

[0076] In order to guide the matching of computing tasks and computing nodes to evolve towards the minimization optimization goal, the intelligent agent should allocate nodes with high computing power to tasks with large computing power as much as possible. Therefore, in the process of modeling the entire deep reinforcement learning environment, the execution time of a single task cannot be simply used to measure the pros and cons of the task scheduling results, but the time consumed by the unit computing amount during the task calculation process is used as the reward function. The reward function of this embodiment is the value obtained by dividing the computing amount corresponding to a single task by the survival time of the task in the system. In a mathematical sense, it represents the computing speed of a single task. The faster the calculation speed, the more the task tends to be executed and completed as soon as possible.

[0077] S22. Formulate task scheduling strategy:

[0078] In the task scheduling strategy, a task j to be executed at t arrive When the task scheduling side is reached at the time, when the computational amount of the task has been determined, this embodiment calculates the expected total computational time T of j. over For discussion, it is necessary to consider the transmission delay T of task information between nodes. trans , the queuing delay T of the task on the assigned computing node wait and execution time T exe . In terms of transmission delay, if j is not compressed in the denoising calculation process, only the transmission delay of the information corresponding to j needs to be considered. Otherwise, the transmission delay of the intermediate vector of the historical task corresponding to j, which is used as the input of the compressed denoising calculation process, needs to be considered. When the amount of transmitted data is M, this embodiment determines the transmission delay between two different computing nodes N and N' through r(N) and r(N'). Transmission delay only occurs when transmitting between different computing nodes, and the delay of data transmission between containers on the same computing node is 0.

[0079]

[0080] When task j is matched to the corresponding computing node N in this embodiment, this embodiment needs to obtain the resource status of the computing node. If the remaining resources of the current computing node can execute the task, there is no queuing delay. If the remaining resources of the current computing node are insufficient to support the execution of task j, all tasks on the computing node are obtained and arranged in descending order by the estimated completion time. The queuing delay of task j is the earliest released. After the release, the remaining resources of the computing node can support the completion time t of task j' of the current computing task j. end The difference between the current time t and the current time t. In addition, the queuing delay is also related to the transmission delay. Even if the currently assigned computing node can immediately perform the calculation of task j, it is necessary to wait for all the content required to execute task j to be transmitted to the computing node. Through the transmission delay and queuing delay, the time t when task j starts to execute can be determined. start .

[0081]

[0082] t start (j) = t arrive (j)+T wait (j,N)

[0083] The computational effort of task j is proportional to the corresponding number of denoising steps s, denoted by C. The computational rate r of all tasks of the load on a computing node in each time interval [t, t+1] is constant. However, the computational rates of multiple computing tasks on a single computing node will affect each other. The relationship between the computational rates of multiple computing tasks in a time interval on a single computing node is expressed as:

[0084]

[0085] This aspect must be considered when calculating the execution time of the current task j. The execution time of task j is T exe , start execution time t start and the execution end time t end The relationship between them is expressed as:

[0086] t end =t start +T exe (j,N)

[0087]

[0088] In summary, the total computation time of task j is T over Statements such as:

[0089] T over (j,N)=T wait (j,N)+T end (j,N)

[0090] For the entire system, the optimization objectives for multiple tasks of the same type are expressed as follows:

[0091]

[0092] S23. Build a task scheduling model:

[0093] A policy network P and a value network V are used to model the matching scheduling of computing tasks and computing nodes, and a proximal policy optimization algorithm is used to train the policy network and the value network.

[0094] The policy network outputs the probability transfer distribution d of the action at the current state st, and performs multiple consecutive actions according to d, corresponding to scheduling the computing task to be executed at the current moment to the computing node, and then updates the system state after each action, and outputs a new state st'. After each task j is completed, the intelligent body will obtain the execution reward R(j) of the task. In this embodiment, the starting state, action, post-execution state and execution reward during the execution of the task are encapsulated as experience q(j) = {st(j), A(j), st'(j), R(j)} and put into the experience pool Q. This is an optimization method based on policy gradients, which maximizes the expected cumulative reward by iteratively updating the strategy. The proximal policy optimization algorithm replays historical strategies through the experience pool for learning and optimization to improve data efficiency and reduce the correlation between samples, making the learning process more stable. V uses the value function to evaluate the value of the action according to the action output by the policy network, and updates the action strategy.

[0095] Directly applying the policy gradient algorithm will lead to too many policy changes. The proximal policy optimization algorithm solves this problem by introducing a proxy objective function. This proxy objective function adds terms to limit the magnitude of policy changes when updating the policy. As the proportion of policy changes, AD is the advantage function of action A, and the probability of selecting action A needs to be calculated under a certain policy and state. The clipping function limits the first parameter to between the second and third parameters, predefines a small value θ as the allowable error range, and sets the small value interval [1-θ,1+θ]. The objective function takes the smaller value between the original policy gradient objective function and the term that limits the magnitude of policy changes and expects to maximize the objective function. In this way, when the magnitude of policy changes exceeds the predefined threshold θ, it will weaken the policy update, thereby ensuring that the policy changes are within a reasonable range and improving the stability of learning.

[0096] In this embodiment, 200 rounds of training are performed on the step number compression prediction model using 6000 pairs of prompt words. Figure 5 (a) shows the trend of loss function results during training. Figure 5(b) shows the use of 1,000 images with different compression step strategies, setting the number of denoising steps in the original image reasoning process to 20, and the compression range from 1 step to 14 steps. The average score of the compressed output image judged by ImageReward is compared with the average score of the output image under the original denoising step number. The experimental results show that under the step compression prediction model, the overall deviation of the prediction score is small.

[0097] For the task model, this embodiment trains 40,000 rounds. During the training process, the sum of the policy loss, value loss and entropy loss is used to update the overall model parameters. Figure 6 Results are shown for the three categories of losses mentioned above and for the overall loss.

[0098] This embodiment evaluates the cases where the cloud computing nodes account for 10% and 20% of the total number of nodes, respectively. The benchmark measurement time is 100 seconds, and the cases where 1 inference request and 2 inference requests are accepted per second are tested. This leads to the following four cases. During the training process, this embodiment alternately trains a total of 50,000 rounds with these four cloud edge resource ratios. During the test process, this embodiment tests each case for 500 rounds.

[0099] One task arrives per second, 10% of which are cloud computing nodes. Figure 7 (a) shows the reward for this situation during training. Figure 7 As shown in (a), the average task completion time obtained by the model of this embodiment is 0.883s, which is shortened by 1.065s, 1.014s and 0.863s respectively compared with the previous three baselines.

[0100] One task arrives per second, 20% of which are cloud computing nodes. Figure 7 (b) shows the reward for this situation during training. Figure 7 As shown in (b), the average job completion time obtained by the model of this embodiment is 0.846s, which is shortened by 1.117s, 1.029s and 0.131s respectively compared with the previous three baselines.

[0101] 2 tasks arrive per second, 10% of which are cloud computing nodes. Figure 7 (c) shows the reward for this situation during training. Figure 7 As shown in (c), the average task completion time obtained by the model is 0.850s, which is 1.116s, 1.033s and 0.092s shorter than the previous three baselines respectively.

[0102] 2 tasks arrive per second, 20% of which are cloud computing nodes. Figure 7 (d) shows the reward for this situation during training. Figure 7As shown in (d), the average task completion time obtained by the model of this embodiment is 0.887s, which is shortened by 1.071s, 1.020s and 0.838s respectively compared with the first three baselines.

[0103] Experiments show that the average task execution time of this framework is lower than that of local execution, greedy allocation and ordinary deep reinforcement learning task scheduling strategies, and it can guarantee image quality during the inference process.

[0104] Therefore, the present invention adopts the above-mentioned diffusion model reasoning acceleration system and method based on model learning, and proposes a novel method for modeling the diffusion reasoning pipeline for the diffusion model which is more common in the current field of text graphs. By sharing the denoising steps between diffusion model reasoning tasks with similar input text semantics and using semantic consistency to achieve step number clipping, a diffusion model reasoning acceleration architecture guided by the step number clipping principle is introduced, which includes a step clipping strategy based on a multi-head attention mechanism. In addition, considering the available computing resources, a task scheduling strategy is designed using deep reinforcement learning. Minimize the task completion time while ensuring image quality.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A diffusion model reasoning acceleration system based on model learning, characterized in that: include: The step number clipping module is used to compress the calculation steps of the denoising calculation process in the diffusion model inference; The task node matching module is used to allocate and schedule computing tasks to the computing nodes of the cloud-edge collaborative network system.

2. According to the diffusion model reasoning acceleration system based on model learning in claim 1, it is characterized in that: Diffusion model reasoning includes: The encoding process maps the prompt word and seed provided by the user to the feature vector of the latent space. The semantic feature vector corresponding to the prompt word is represented by p, and the seed feature vector is represented by s. p and s are used together as the input of the denoising process. The denoising calculation will be iterated n times, and after each step i, an intermediate vector s with the same dimension as the seed feature vector is output i , and use the intermediate vector and p as the input vector for the next step of denoising calculation; The decoding process is to convert the intermediate vector s output from the last step of the denoising calculation process n Mapping from latent vector space to image space is provided to users.

3. A method for applying a diffusion model reasoning acceleration system based on model learning as claimed in claim 1, characterized in that: The following steps are involved: S1, according to the step number clipping module, when the denoising calculation unit in the diffusion model reasoning performs calculation, by formulating a step compression strategy and building a step number compression prediction model, the denoising calculation steps are compressed; S2. Classify the computing tasks through the step compression model, determine the execution computing amount and input of the task, and then allocate and schedule the computing tasks to the computing nodes of the cloud-edge collaborative network system according to the task node matching module.

4. The method of a diffusion model reasoning acceleration system based on model learning according to claim 2, characterized in that: The specific steps of S1 are as follows: S11, formulate a step compression strategy, by saving the intermediate vectors of user requests in the reasoning process, when a new user request arrives, if there is a prompt word in the historical user request whose similarity with the prompt word of the new user request reaches a preset threshold, the denoising calculation of the current user request is started from the j+1th step, and the jth intermediate vector of the historical user request that meets the requirements and the feature vector of the current user prompt word are selected as the input of the denoising calculation, and the denoising calculation of the current user request is compressed from n steps to nj steps; S12. Build a step compression prediction model based on the multi-head attention mechanism. Before model training, obtain the number of inference steps n, the feature vector s of a specific seed, and the feature vectors p1 and p2 corresponding to two prompt words whose similarity exceeds the threshold k. Use the cosine similarity cos() to measure the similarity between p1 and p2. The selected prompt word pair satisfies the condition cos(p1,p2)>k. For p1 and p2 that meet the conditions, first perform inference on p1 and p2 on n and s respectively and save the intermediate vector group S = {s1,s2,…,s n }, and then use n, p2 and each vector in the intermediate vector group to perform {n-1, n-2, …, 1} steps of denoising calculation respectively, decode the image, and then use the ImageReward scoring framework to score the original image of p2 and the image output by the compression step, and organize the scores into a vector I; during the model training process, s, p1, p2 and I are input, the training model predicts I through the information of s, p1, p2, and uses Smooth L1 loss to guide the update of model parameters.

5. The method of a diffusion model reasoning acceleration system based on model learning according to claim 3, characterized in that: The specific steps of S2 are as follows: S21. Model the task scheduling environment, including state space, JCDRL, action space and environment; S22. Formulate a task scheduling strategy to obtain the total computing time T of computing task j. over and the optimization objective expressions under several similar tasks; S23. Use a policy network P and a value network V to model a task scheduling model for matching computing tasks with computing nodes.

6. The method of a diffusion model reasoning acceleration system based on model learning according to claim 5, characterized in that: The state space in S21 is defined by the states of the computing nodes N and the surviving tasks J in the system. The surviving tasks include the tasks to be assigned to computing nodes and the tasks being executed. The action space A(t) indicates that each task entering the system in the interval [t, t+1] is sequentially scheduled to the computing nodes and updates the status of the surviving tasks and computing nodes in the entire system; In the cloud-edge network system, an intelligent agent is used to perform task scheduling actions, which triggers the state transition of the entire cloud-edge network system. The transition function is: Pr[ST(t+1)=st'|ST(t)=st,A(t)] ST(t) represents the current execution status of the task being executed; t represents the time; ST(t+1) represents the updated state of the state space after the agent completes all actions at time t; st represents the current state, and st' represents the new state output after the system state is updated.

7. The method of a diffusion model reasoning acceleration system based on model learning according to claim 5, characterized in that: The specific contents of S22 are: In the task scheduling strategy, the task j to be executed is arrive When the task scheduling side is reached at the time, when the computational amount of the task has been determined, the expected total computational time T of the computation j is calculated. over , including the transmission delay T of task information between nodes trans , the queuing delay T of the task on the assigned computing node wait and execution time T exe ; For the transmission delay, if j is not compressed in the denoising calculation process, the transmission delay of the information corresponding to j is selected. Otherwise, the transmission delay of the intermediate vector of the historical task corresponding to j used as the input of the compressed denoising calculation process is also selected. When the amount of transmitted data is M, the transmission delay between two different computing nodes N and N' is determined by r(N) and r(N'): When task j matches the corresponding computing node N, obtain the resource status of the computing node. If the remaining resources of the current computing node can execute the task, there is no queuing delay. If the remaining resources of the current computing node are not enough to support the execution of task j, obtain all tasks on the computing node and arrange them in descending order according to the expected completion time. The queuing delay of task j is the earliest release. After the release, the remaining resources of the computing node support the completion time t of task j' of the current computing task j. end The difference between the current time t and the time t at which task j starts execution is determined by transmission delay and queuing delay. start , queuing delay and the time t at which task j starts execution start The expression is as follows: t start (j)=t arrive (j)+T wait (j,N) The computational amount of task j is proportional to the corresponding number of denoising steps s, denoted by C; the computational rate r of all tasks of the load on the computing node in each time interval [t, t+1] is constant. The relationship between the computational rates of multiple computing tasks in the time interval on a single computing node is expressed as: The execution time T of task j exe , start execution time t start and the execution end time t end The relationship between them is expressed as: t end =t start +T exe (j,N) In summary, the total computation time of task j is T over The expression is as follows: T over (j,N)=T wait (j,N)+T end (j,N) The optimization objective expression for multiple tasks of the same type is as follows:

8. The method of a diffusion model reasoning acceleration system based on model learning according to claim 5, characterized in that: The specific contents of S23 are: The policy network outputs the probability transfer distribution d of the action at the current state st, and executes multiple consecutive actions based on d, corresponding to scheduling the computing task to be executed at the current moment to the computing node, and then updates the system state after each action, outputting a new state st'; for each task j, after the completion of the task, the agent obtains the execution reward R(j) of the task, and encapsulates the starting state, action, post-execution state and execution reward during the execution of the task into experience q(j) = {st(j), A(j), st'(j), R(j)} and puts it into the experience pool Q; V uses the value function to evaluate the value of the action according to the action output by the policy network, and updates the action strategy.

Images