Diffusion model edge-end collaborative reasoning method and device based on migration reinforcement learning

Abstract

The invention discloses a diffusion model edge-end collaborative reasoning method and device based on migration reinforcement learning, and the method comprises the steps: constructing an image task processing system which comprises a plurality of user terminals and edge equipment, and the edge equipment comprises a main node and a plurality of auxiliary nodes; describing a task unloading decision optimization problem received by the main node as a Markov decision process, wherein the Markov decision process comprises a state space, an action space and a reward function; constructing a task unloading decision model at the main node based on a DDQN network; the main node trains the task unloading decision model, and generates an optimal unloading decision according to the trained task unloading decision model; and the main node sends the received task to the corresponding edge and / or user terminal according to the optimal unloading decision, and the edge and / or local terminal processes the task by using an image generation model constructed based on the U-ViT model. According to the method, intelligent dynamic allocation of the tasks between the local and the edge is realized, and the resource utilization rate is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a diffusion model edge collaborative reasoning method and device based on transfer reinforcement learning. Background Art

[0002] Diffusion models are an important breakthrough in the field of artificial intelligence in recent years, showing great potential in many fields such as image generation, speech synthesis, and molecular design. This type of model generates high-quality samples by simulating the process of gradual noise removal, and can produce extremely realistic and diverse outputs. With large-scale pre-training and architecture optimization, diffusion models have surpassed many traditional methods in terms of generation quality and diversity, becoming one of the hot topics in current generative model research.

[0003] Despite the excellent performance of diffusion models, their practical application still faces major challenges. The primary problem is the high computational complexity, which usually requires powerful hardware support, which limits its deployment on resource-constrained edge devices. Currently, the reasoning of diffusion models mainly relies on cloud servers, which brings privacy and security risks, strong network dependence, high latency, and high operating costs. Although academia and industry have begun to explore edge computing solutions, it is still difficult to directly deploy complex diffusion models on edge devices due to the limited computing power of edge devices.

[0004] To solve the above problems, edge computing and transfer learning technology provide an innovative solution, namely device edge collaborative reasoning. This method allows the diffusion model to be divided into two stages, and then these tasks are intelligently distributed between mobile devices and edge computing infrastructure to achieve balanced computing load. At the same time, the cost of redeployment and training can be effectively reduced through transfer learning. This method can not only improve efficiency and reduce costs while ensuring the quality of reasoning, but also effectively balance privacy protection and performance requirements.

[0005] In VR / AR immersive scene generation, diffusion models can achieve high-quality virtual environment rendering, but their deployment faces the following challenges:

[0006] 1. Unbalanced allocation of computing resources: Existing edge-cloud collaboration methods use a fixed scene rendering task allocation strategy and cannot flexibly schedule computing resources based on user interaction needs and network conditions. For example, when multiple users perform complex VR interactions at the same time, some edge nodes may experience serious computing backlogs, while other nodes are in a resource-idle state, resulting in low overall rendering efficiency.

[0007] 2. The decision-making ability of offloading strategies based on reinforcement learning is low: Existing reinforcement learning methods have difficulty in accurately evaluating and predicting complex user interaction patterns and network fluctuations when dealing with dynamic task offloading in VR / AR scenarios. Especially in scenarios with multiple users accessing concurrently and frequently changing network conditions, reinforcement learning models often make suboptimal offloading decisions, resulting in rendering delays and screen freezes, affecting user experience.

[0008] 3. Insufficient migration capability of offloading strategies: When migrating a task offloading strategy trained in a specific VR / AR application scenario to a new application scenario, the strategy performance significantly decreases due to differences in user interaction patterns, network environments, and computing load characteristics. Existing methods are difficult to effectively extract and generalize common features in different scenarios, resulting in the need to retrain the offloading strategy every time a new application is deployed, greatly increasing deployment costs. Summary of the invention

[0009] In response to the above problems, the present invention proposes a diffusion model edge collaborative reasoning method and device based on transfer reinforcement learning, which provides a new technical route for the edge deployment and application of the diffusion model, and is expected to promote the practical application of the diffusion model in a wider range of scenarios. To solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0010] A diffusion model edge collaborative reasoning method based on transfer reinforcement learning includes the following steps:

[0011] S1, constructing an image task processing system including several user terminals and edge devices, wherein the edge devices include a main node and several auxiliary nodes;

[0012] S2, describing the offloading decision optimization problem of the image processing task received by the master node as a Markov decision process, wherein the Markov decision process includes a state space, an action space, and a reward function;

[0013] S3, builds a task offloading decision model on the master node based on the DDQN network;

[0014] S4, the master node trains the task offloading decision model constructed in step S3 according to the state space, action space and reward function obtained in step S2, and generates an optimal offloading decision according to the trained task offloading decision model;

[0015] S5, the master node sends the received task to the corresponding edge device and / or user terminal according to the optimal offloading decision, and the edge device and / or local terminal respectively processes the task using the image generation model constructed based on the U-ViT model.

[0016] In step S2, the state space adopts S n express, S n represents the state space observed by the nth user terminal in the image task processing system, The metadata of the task on the nth user terminal, represents the computational condition of the edge device observed by the nth user terminal, represents the communication state of the channel observed by the nth user terminal;

[0017] The action space adopts A n express, A n represents the action space of the nth user terminal, represents the binary offloading decision of the task on the nth user terminal, Indicates the selection status of the offloading server for the task on the nth user terminal, represents the number of denoising steps to be performed by the local terminal when processing the task on the nth user terminal;

[0018] The expression corresponding to the reward function is:

[0019]

[0020] In the formula, R n,t represents the reward function of the nth user terminal at time t, represents the total inference time of the image processing task generated by the nth user terminal at time t after being offloaded, Indicates the total inference time taken by the nth user terminal at time t if the image processing task generated by the nth user terminal is run completely locally.

[0021] In step S3, the task offloading decision model includes a Q network and a target Q network, and both the Q network and the target Q network include a state encoding module for receiving a state space and optimizing a Q value, a time encoding module for receiving a time step, a fully connected layer for splicing the output of the state encoding module and the output of the time encoding module, an activation function layer for calculating the Q value, a denoising module for optimizing the Q value using a diffusion model, and an action optimization module for selecting an action based on the optimized Q value.

[0022] The step S4 comprises the following steps:

[0023] S4.1, input the state space obtained in step S2 into the Q network, select the action based on the ε greedy strategy to obtain the corresponding reward and the state at the next moment, and convert (s t ,a t ,r t ,s t+1 ) is stored in the experience buffer, where s trepresents the state at time t, a t represents the action at time t, r t represents the reward at time t, s t+1 Indicates the state at time t+1;

[0024] S4.2, randomly draw N samples from the experience buffer;

[0025] S4.3, initialize the number of training rounds τ = 1, and randomly initialize the parameters θ of the Q network τ , initialize the parameters of the target Q network

[0026] S4.4, set the diffusion step K, and initialize the current time step k=K;

[0027] S4.5, the state encoding module is based on the Q value and state s output by the denoising module t Generate state feature vector. The time encoding module generates a time feature vector based on the current time step k. The fully connected layer concatenates the state feature vector and the time feature vector.

[0028] S4.6, the activation function layer calculates the Q value based on the concatenated feature vector output by the fully connected layer;

[0029] S4.7, determine k≤1, if yes, k=K, execute step S4.8, otherwise, the denoising module optimizes the Q value based on the Q value obtained in step S4.6 using the diffusion model, execute k=k-1, input the optimized Q value into the state coding module and return to step S4.5;

[0030] S4.8, the action optimization module selects an action based on the maximum Q value, and the target Q network calculates the Q value of the next state based on the selected action;

[0031] S4.9, calculating the target Q value based on the Q value of the next state obtained in step S4.8;

[0032] S4.10, calculate the loss function, and update the Q network with the goal of minimizing the loss function;

[0033] S4.11, determine τ%Γ target = 0, if so, update the target Q network based on the Q network parameters, that is, every training Γ target round, copy the parameters of the Q network to the target Q network once, and execute step S4.12, otherwise, directly execute step S4.12, where Γ target Indicates the update frequency of the target Q network;

[0034] S4.12, determine τ%Ψ train =0, if yes, the training is finished, otherwise, execute τ=τ+1 and return to step S4.5, where Ψtrain Represents the total number of training rounds.

[0035] Before executing step S4.9, determine If yes, execute the diffusion step cycle update decision and then execute step S4.9. Otherwise, execute step S4.9 directly, where: represents the diffusion step update period;

[0036] The method for executing the diffusion step cycle update decision is: If yes, execute K=K-1 and update the diffusion step K. Otherwise, execute K=K+1 and update the diffusion step K. Indicates The average Q value within the diffusion step update cycle, Indicates The average Q value within the diffusion step update cycle, δ represents the Q value difference threshold;

[0037] The expression of the updated diffusion step K is:

[0038]

[0039] In the formula, K min represents the preset minimum diffusion step, K max Indicates the preset maximum diffusion step.

[0040] Before training the task offloading decision model, the fully connected layer of the task offloading decision model is randomly initialized and the parameters of other network layers are frozen.

[0041] In step S5, the image generation model includes an image embedding layer, several Transformer blocks, an output processing layer and a quality discrimination module connected in sequence, and the quality discrimination module is also connected to the image embedding layer; the quality discrimination module uses a preset discrimination network to output the visual quality score of the intermediate image generated by the output processing layer, and compares the visual quality score with the visual quality threshold, and determines whether to output the intermediate image as the final image based on the comparison result.

[0042] A diffusion model edge collaborative reasoning device based on transfer reinforcement learning includes several user terminals and edge devices, wherein the edge devices include a main node and several auxiliary nodes, and the main node is provided with:

[0043] Data collection module: used to describe the offloading decision optimization problem of the image processing task received by the master node as a Markov decision process, wherein the Markov decision process includes a state space, an action space and a reward function;

[0044] Decision model building module: used to build a task offloading decision model based on the DDQN network;

[0045] Decision optimization module: used to train the task offloading decision model constructed by the model construction module based on the state space, action space and reward function obtained by the data collection module, generate the optimal offloading decision according to the trained task offloading decision model, and send the optimal offloading decision to the corresponding edge device and / or user terminal;

[0046] Image processing module I: used to process the task using an image generation model based on the optimal offloading decision generated by the decision optimization module, wherein the image generation model is constructed based on the U-ViT model;

[0047] The user terminal is provided with:

[0048] Image processing module II: used to process the task using an image generation model based on the optimal offloading decision generated by the decision optimization module, wherein the image generation model is constructed based on the U-ViT model.

[0049] Beneficial effects of the present invention:

[0050] 1. Integrate individual rationality and environmental dynamics: By improving the existing reinforcement learning technology, the intelligent dynamic allocation of computing tasks between local devices and edge terminals is realized, and adaptive adjustments are made according to real-time network conditions and device loads, significantly improving resource utilization efficiency. At the same time, the designed DDQN algorithm based on decision diffusion can adaptively select the appropriate diffusion step size, thereby achieving a better balance between decision quality and decision speed;

[0051] 2. Dynamic exit model improves reasoning efficiency: We designed and implemented an image generation model with a dynamic exit judgment mechanism, which can dynamically judge the current image quality during the reasoning process, thereby significantly improving computational efficiency. Through dynamic judgment, the model enables the reasoning process to exit early when a specific accuracy standard is met, reducing unnecessary computational overhead;

[0052] 3. Efficient deployment and training: The migration algorithm significantly speeds up the redeployment of the model and reduces the consumption of computing resources. It avoids the high cost of retraining all parameters while improving the adaptability and stability of the model in the new environment. By retaining previously learned knowledge, it not only reduces the overall training cost, but also enhances the generalization ability of the model in a variety of practical applications, thereby improving the overall performance of the deep learning framework in edge task offloading. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A schematic diagram of the structure of the image generation model;

[0055] Figure 2 It is a schematic diagram of the process of the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of the Q network. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] A diffusion model edge collaborative reasoning method based on transfer reinforcement learning, such as Figure 2 As shown, the following steps are included:

[0059] S1, constructing an image task processing system including several user terminals and edge devices, wherein the edge devices include a main node and several auxiliary nodes;

[0060] Each user terminal is connected to a corresponding auxiliary node, and all auxiliary nodes are connected to the main node. When the user sends an image processing task through the user terminal, the task is first transmitted to the main node through the corresponding auxiliary node. The main node outputs the task offloading decision based on the task offloading decision model set thereon, and then the corresponding node or user terminal, that is, the local terminal, processes the task based on the task offloading decision.

[0061] S2, describing the offloading decision optimization problem of the image processing task received by the master node as a Markov decision process (MDP), wherein the Markov decision process includes a state space, an action space, and a reward function;

[0062] The state space is S n express, S n represents the state space observed by the nth user terminal in the image task processing system, Indicates the metadata of the current task on the nth user terminal, the metadata including the number of image denoising steps, image resolution requirements, and local device computing capabilities, represents the computing conditions of the edge device observed by the nth user terminal, wherein the computing conditions include edge computing capability, CPU occupancy level, and GPU occupancy level. Indicates the communication state of the channel observed by the nth user terminal. All user terminals send the observed state spaces to the master node respectively, and the master node generates corresponding actions based on all received state spaces.

[0063] The action space adopts A n express, A n represents the action space of the nth user terminal, represents the binary offloading decision of the task on the nth user terminal, that is, whether to offload the task on the nth user terminal to the edge device or the local terminal, or to the edge device and the local terminal. Indicates the selection status of the server for offloading the task on the nth user terminal, that is, to which edge nodes the image processing task on the nth user terminal is offloaded. Indicates the number of denoising steps that the local terminal needs to perform when processing the current image processing task on the nth user terminal.

[0064] The reward function uses R n The corresponding expression is:

[0065]

[0066] In the formula, R n,t represents the reward function of the nth user terminal at time t, represents the total inference time of the image processing task generated by the nth user terminal at time t after being offloaded, It represents the total inference time taken by the image processing task generated by the nth user terminal at time t if it is completely run locally. The calculation of the above two total inference time data is prior art and will not be described in detail in this embodiment. According to formula (1), if the total task time after offloading exceeds the maximum tolerance value, it means that this is a bad offloading decision. The maximum tolerance value can be set as the time taken by the image processing task to be completely inferred in the local terminal.

[0067] S3, constructing a task offloading decision model on the master node based on the DDQN (Double Deep Q-Network) network, wherein the task offloading decision model includes a Q network and a target Q network, wherein the Q network and the target Q network both include a state encoding module for receiving a state space and optimizing a Q value, a time encoding module for receiving a time step, a fully connected layer for splicing an output of the state encoding module and an output of the time encoding module, an activation function layer for calculating the Q value, a denoising module for optimizing the Q value using a diffusion model, and an action optimization module for selecting an action based on the optimized Q value;

[0068] Since the collaborative reasoning service problem of the master node is a target optimization problem with coupling constraints, its core is the problem of task offloading in a dynamic environment. In order to solve the uncertainty in the decision-making process, this application proposes a learning algorithm based on deep reinforcement learning, which will be used to make offloading decisions for the current image processing task in a dynamic environment, including the selection of the server, i.e., the edge device and / or the local terminal, and the amount of offloaded tasks, i.e., the number of denoising steps performed by the corresponding server.

[0069] The task offloading decision model of this application adopts the DDQN framework, which includes a policy network, namely the Q network, a target network, namely the target Q network, and an experience buffer. The policy network interacts with the environment and records the state space and action space as transitions (S n ,A n ,R t ,R t+1 ) are stored in the experience buffer, and the algorithm uses these stored transfers to optimize the network. The forward process of the Q network is replaced by a decision generation process based on a diffusion model. Studies have shown that the diffusion process is effective in policy networks because it has a strong ability to capture the dependencies between observations and actions.

[0070] The network structure of the Q network is the same as that of the target Q network, and the following description is based only on the Q network. Figure 3 As shown, in order to further improve the state perception ability of the policy network in the diffusion process, the present application constructs a lightweight attention-based state encoding module in the input stage. Specifically, in the policy network, the output end of the state encoding module and the output end of the time encoding module are both connected to the input end of the fully connected layer, the output end of the fully connected layer is connected to the input end of the activation function layer, the output end of the activation function layer is connected to the input end of the denoising module, the output end of the denoising module is connected to the input end of the action optimization module and the input end of the state encoding module, and the output end of the action optimization module is connected to the input end of the state encoding module of the target Q network.

[0071] The state encoding module encodes the observed state and the optimized Q value output by the denoising module into a state feature vector suitable for model processing, which is used for subsequent feature extraction and decision-making processes. Compared with the ordinary linear mapping method, this module enables the policy network to better perceive state changes in a small number of diffusion steps. The time encoding module of the Q network encodes the current time step information into a time feature vector suitable for model use, so as to better consider the influence of time in the prediction of the generation process or the reverse diffusion process. The time step is a scalar describing the current diffusion step. The fully connected layer concatenates the state feature vector and the time feature vector and transmits them to the activation function layer. The activation function layer generates the Q value of each action based on the concatenated feature vector output by the fully connected layer. The denoising module uses the diffusion model to denoise the Q value of each action and outputs the Q value of the next time step. At the same time, the Q value of the next time step is input into the state encoding module. The state encoding module, the time encoding module, the fully connected layer, the activation function layer and the denoising module optimize the Q value based on the preset total number of diffusion steps according to the above steps. The action generation module selects an action based on the optimized Q value of each action output by the denoising module. This network structure can combine state and time information for comprehensive processing, thereby improving the accuracy of decision-making and the ability to respond to environmental changes.

[0072] S4, the master node trains the task offloading decision model constructed in step S3 according to the state space, action space and reward function obtained in step S2, and generates an optimal offloading decision according to the trained task offloading decision model, including the following steps:

[0073] S4.1, input the state space obtained in step S2 into the Q network, select the action based on the ε greedy strategy and obtain the corresponding reward and the state at the next moment, and convert (s t ,a t ,r t ,s t+1 ) is stored in the experience buffer, where s t represents the state at time t, that is, the set of state spaces of all user terminals at time t, a t represents the action at time t, i.e., the set of action spaces of all user terminals at time t, r t represents the reward at time t, s t+1 Indicates the state at time t+1;

[0074] S4.2, randomly draw N samples from the experience buffer;

[0075] S4.3, initialize the number of training rounds τ = 1, and randomly initialize the parameters θ of the Q network τ , initialize the parameters of the target Q network

[0076] S4.4, set the diffusion step K, and initialize the current time step k = K;

[0077] K is a positive integer greater than or equal to 2. In this embodiment, K=5.

[0078] S4.5, the state encoding module is based on the Q value and state s output by the denoising module t Generate state feature vector. The time encoding module generates a time feature vector based on the current time step k. The fully connected layer concatenates the state feature vector and the time feature vector.

[0079] When τ=1, the state encoding module randomly generates Gaussian noise as the Q value of the first input.

[0080] S4.6, the activation function layer calculates the Q value based on the concatenated feature vector output by the fully connected layer;

[0081] S4.7, determine k≤1, if yes, k=K, execute step S4.8, otherwise, the denoising module optimizes the Q value based on the Q value obtained in step S4.6 using the diffusion model, execute k=k-1, input the optimized Q value into the state coding module and return to step S4.5;

[0082] The calculation formula of the optimized Q value is:

[0083]

[0084] In the formula, x k represents the Q value at the kth step, represents the noise predicted by the diffusion model, σ k is a term that controls random noise, usually used to increase the diversity of sampling, z represents standard Gaussian noise, β k The injection intensity of the noise at each step is controlled by a predefined parameter, β k The value of grows linearly in the range of [0.001, 0.1] during the diffusion process. In the denoising process, the cumulative form used

[0085] S4.8, the action optimization module selects an action based on the maximum Q value, and the target Q network calculates the Q value of the next state based on the selected action;

[0086] The corresponding expression for the action selection module to select an action based on the maximum Q value is:

[0087]

[0088] S4.9, calculating the target Q value based on the Q value of the next state obtained in step S4.8;

[0089] yt =r t +γQ(s t+1 ,a ′ t θ - ); (4)

[0090] In the formula, γ represents the discount factor, s t+1 represents the state at time t+1, Q(s t+1 ,a ′ t θ - ) indicates the target Q network output based on action a ′ t Q value, y t Represents the target Q value at time t.

[0091] S4.10, calculate the loss function, and update the Q network with the goal of minimizing the loss function;

[0092]

[0093] In the formula, y i represents the target Q value of the i-th sample, Q(s i ,a i ; θ) represents the Q value of the i-th sample, 1≤i≤N.

[0094] S4.11, determine ρ%Γ target = 0, if so, update the target Q network based on the Q network parameters, that is, every training Γ target round, copy the parameters of the Q network to the target Q network once, and execute step S4.12, otherwise, directly execute step S4.12, where Γ target Indicates the update frequency of the target Q network;

[0095] S4.12, determine ρ%Ψ train =0, if yes, the training is finished, otherwise, execute ρ=ρ+1 and return to step S4.5, where Ψ train Represents the total number of training rounds.

[0096] Preferably, during the training of the task offloading decision model, the fluctuation state of the Q value can be evaluated at regular intervals, and the diffusion step K can be adjusted according to the evaluation result. Specifically, before executing step S4.9, determine If it is zero, execute the diffusion step cycle update decision, that is, step ①, and then execute step S4.9. Otherwise, directly execute step S4.9, where: represents the diffusion step update period;

[0097] ①, judge If so, execute K=K-1 and update the diffusion step K based on formula (6). Otherwise, execute K=K+1 and update the diffusion step K based on formula (6), where: Indicates the current diffusion step update cycle, i.e. The average Q value within the diffusion step update cycle, that is, Round, round, ..., round ρ-1, and round ρ in total The mean Q value of the round, Indicates The average Q value within the diffusion step update cycle, that is, Round, Round, ..., Round and Total rounds The mean Q value of the round;

[0098]

[0099] In the formula, K min represents the preset minimum diffusion step, K max Indicates the preset maximum diffusion step. Setting an upper and lower limit here can avoid too many or too few diffusion steps in extreme cases, so that the diffusion step number always fluctuates within a reasonable range.

[0100] The above designed an adaptive diffusion step adjustment mechanism, which estimates the degree of current state change by comparing the mean Q value calculated by the Q network in the current diffusion step update cycle with the mean Q value in the previous diffusion step update cycle, and determines the diffusion step number according to the degree of state change. The following adjustment rules are implemented:

[0101] 1) Increase the number of diffusion steps when there is a significant change: When the state changes significantly (the Q value fluctuates greatly), the number of diffusion steps increases to fully optimize the Q value and ensure that the strategy is optimal in the new state;

[0102] 2) Reduce the number of diffusion steps when there are small changes: When the state changes are small or close to stability (the Q value changes slightly), reduce the number of diffusion steps to avoid repeated calculations;

[0103] 3) Control the diffusion step adjustment according to the preset threshold δ, for example is an update cycle (i.e., taking 10 Q values ​​of the policy network as one cycle and calculating the average once). When the difference between the average Q value in this cycle and the average Q value in the previous cycle is less than δ, the number of diffusion steps is reduced, otherwise it is increased.

[0104] Preferably, before training the task offloading decision model, the fully connected layer of the task offloading decision model is initialized and the parameters of other network layers are frozen. The present application redeploys the task offloading decision model when the environment changes, but only initializes the fully connected layer in the neural network used to encode the input, while the remaining parameters are used as obtained in the previous training. This method retains the knowledge of feature extraction parts such as other layers by selectively initializing a portion of the model's parameters, maintains the continuity of existing knowledge, reduces training time and resource consumption in the new environment, and at the same time improves the model's adaptability in the new environment, thereby speeding up the redeployment process.

[0105] Specifically, when the edge device environment changes, for example, due to differences in network conditions or device performance, the task offloading requirements change, and only some network parameters need to be reinitialized. This is because the fully connected layer is usually responsible for the specific classification or regression operations of the task, and its parameters are more easily affected by environmental changes, while the parameters of other layers are mainly used for the extraction of common features and have good cross-environment adaptability. This method enables the model to be quickly deployed in a new edge environment, which not only saves computing resources but also ensures the effectiveness and accuracy of the model.

[0106] S5, the master node sends the received task to the corresponding edge device and / or user terminal according to the optimal offloading decision, and the edge device and / or local terminal respectively processes the task using the image generation model built based on the U-ViT (Universal Vision Transformer) model;

[0107] In order to achieve efficient network reasoning in the reverse denoising of the diffusion process, one of the most direct and effective methods is to optimize the backbone network itself. Network reasoning in the diffusion process is essentially a regression task. This application chooses to build a quality discrimination module based on the prediction of intermediate results in the model, which can achieve early exit from image processing and reduce computational overhead. This mechanism relies on a lightweight discriminant network that evaluates the quality of the intermediate generated images and determines whether the generation process can be exited early. This method is particularly suitable for real-time generation tasks. By performing quality judgments at different stages of the generation process, efficient use of resources can be achieved.

[0108] like Figure 1As shown, the image generation model includes an image embedding layer, several Transformer blocks, an output processing layer and a quality discrimination module connected in sequence, and the quality discrimination module is also connected to the image embedding layer. The image embedding layer is used to receive image processing tasks, divide the received image into image blocks (usually called patches), and map each patch to a specific feature space through a linear transformation. This process usually converts the two-dimensional image data into a one-dimensional embedding form to output the linear embedding of these patches. The Transformer block processes the embedding sequence output by the image embedding layer, captures the global and local feature relationships through a multi-head self-attention mechanism and a feedforward neural network, and outputs a feature representation processed and normalized by the attention mechanism. The output processing layer maps the final feature representation output by the Transformer block to the final output form, such as reconstructing an image, generating predicted noise, etc. It usually includes additional convolutional layers to improve the quality of visual output. In the image generation model, except for the quality discrimination module, all others are prior art, and this application will not be elaborated in detail.

[0109] The quality discrimination module can be designed into a lightweight discriminant network for denoising the image. The embodiment of the present application adopts a two-layer convolutional neural network, which can also be used to determine whether the intermediate image generated by the output processing layer has achieved a visually realistic effect. The discriminant network is pre-trained with images and their corresponding visual quality scores, and can quickly evaluate the visual features of the image such as overall quality, clarity, structural integrity, etc. The training of the discriminant network is a prior art and is not described in detail in this application. The output of the discriminant network is a visual quality score, which reflects the realism (i.e., visual quality) of the current intermediate image. A visual quality threshold is also preset in the discriminant network, reflecting the minimum quality standard that needs to be achieved in generating the image. The threshold can be adjusted according to the requirements of the task to meet different quality requirements and resource consumption balance. When the output processing layer generates an intermediate image each time, the quality discrimination module compares the visual quality score output by the discrimination network with the visual quality threshold. If the visual quality score is less than the visual quality threshold, the quality discrimination module inputs the generated intermediate image into the image embedding layer, and continues to process the image using the image embedding layer, Transformer block and output processing layer, that is, continues to enter the next stage of denoising, thereby gradually improving the image quality. If the visual quality score is higher than the visual quality threshold, it is considered that the image has achieved a sufficiently realistic effect, and early exit is triggered immediately, stopping the subsequent generation process, and directly outputting the generated intermediate image as the final image, and the image processing task is completed.

[0110] The embodiment of the present application also provides a diffusion model edge collaborative reasoning device based on transfer reinforcement learning, including several user terminals and edge devices, wherein the edge devices include a main node and several auxiliary nodes, and the main node is provided with:

[0111] Data collection module: used to describe the offloading decision optimization problem of the image processing task received by the master node as a Markov decision process, wherein the Markov decision process includes a state space, an action space and a reward function;

[0112] Decision model building module: used to build a task offloading decision model based on the DDQN network;

[0113] Decision optimization module: used to train the task offloading decision model constructed by the model construction module based on the state space, action space and reward function obtained by the data collection module, generate the optimal offloading decision according to the trained task offloading decision model, and send the optimal offloading decision to the corresponding edge device and / or user terminal;

[0114] Image processing module I: used to process the task using an image generation model based on the optimal offloading decision generated by the decision optimization module, wherein the image generation model is constructed based on the U-ViT model;

[0115] The user terminal is provided with:

[0116] Image processing module II: used to process the task using an image generation model based on the optimal offloading decision generated by the decision optimization module, wherein the image generation model is constructed based on the U-ViT model.

[0117] An embodiment of the present application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned diffusion model edge collaborative reasoning method based on transfer reinforcement learning are performed.

[0118] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned diffusion model edge collaborative reasoning method based on transfer reinforcement learning are executed. Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the above-mentioned embodiment of the diffusion model edge collaborative reasoning method based on transfer reinforcement learning.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A diffusion model edge collaborative reasoning method based on transfer reinforcement learning, characterized in that: The steps include: S1, constructing an image task processing system including several user terminals and edge devices, wherein the edge devices include a main node and several auxiliary nodes; S2, describing the offloading decision optimization problem of the image processing task received by the master node as a Markov decision process, wherein the Markov decision process includes a state space, an action space, and a reward function; S3, builds a task offloading decision model on the master node based on the DDQN network; S4, the master node trains the task offloading decision model constructed in step S3 according to the state space, action space and reward function obtained in step S2, and generates an optimal offloading decision according to the trained task offloading decision model; S5, the master node sends the received task to the corresponding edge device and / or user terminal according to the optimal offloading decision, and the edge device and / or local terminal respectively processes the task using the image generation model constructed based on the U-ViT model.

2. The edge collaborative reasoning method of the diffusion model based on transfer reinforcement learning according to claim 1 is characterized in that: In step S2, the state space adopts... n express, S n represents the state space observed by the nth user terminal in the image task processing system, The metadata of the task on the nth user terminal, represents the computational condition of the edge device observed by the nth user terminal, represents the communication state of the channel observed by the nth user terminal; The action space adopts A n express, A n represents the action space of the nth user terminal, represents the binary offloading decision of the task on the nth user terminal, Indicates the selection status of the offloading server for the task on the nth user terminal, represents the number of denoising steps to be performed by the local terminal when processing the task on the nth user terminal; The expression corresponding to the reward function is: In the formula, R n,t represents the reward function of the nth user terminal at time t, represents the total inference time of the image processing task generated by the nth user terminal at time t after being offloaded, Indicates the total inference time taken by the nth user terminal at time t if the image processing task generated by the nth user terminal is run completely locally.

3. The edge collaborative reasoning method of the diffusion model based on transfer reinforcement learning according to claim 1 is characterized in that: In step S3, the task offloading decision model includes a Q network and a target Q network, and both the Q network and the target Q network include a state encoding module for receiving a state space and optimizing a Q value, a time encoding module for receiving a time step, a fully connected layer for splicing the output of the state encoding module and the output of the time encoding module, an activation function layer for calculating the Q value, a denoising module for optimizing the Q value using a diffusion model, and an action optimization module for selecting an action based on the optimized Q value.

4. The edge collaborative reasoning method of the diffusion model based on transfer reinforcement learning according to claim 3 is characterized in that: The step S4 comprises the following steps: S4.1, input the state space obtained in step S2 into the Q network, select the action based on the ε greedy strategy to obtain the corresponding reward and the state at the next moment, and convert (s t ,a t ,r t ,s t+1 ) is stored in the experience buffer, where s t represents the state at time t, a t represents the action at time t, r t represents the reward at time t, s t+1 Indicates the state at time t+1; S4.2, randomly draw N samples from the experience buffer; S4.3, initialize the number of training rounds τ = 1, and randomly initialize the parameters θ of the Q network τ , initialize the parameters of the target Q network S4.4, set the diffusion step K, and initialize the current time step k = K; S4.5, the state encoding module is based on the Q value and state s output by the denoising module t Generate state feature vector. The time encoding module generates a time feature vector based on the current time step k. The fully connected layer concatenates the state feature vector and the time feature vector. S4.6, the activation function layer calculates the Q value based on the concatenated feature vector output by the fully connected layer; S4.7, determine k≤1, if yes, k=k, execute step S4.8, otherwise, the denoising module optimizes the Q value based on the Q value obtained in step S4.6 using the diffusion model, execute k=k-1, input the optimized Q value into the state coding module and return to step S4.5; S4.8, the action optimization module selects an action based on the maximum Q value, and the target Q network calculates the Q value of the next state based on the selected action; S4.9, calculating the target Q value based on the Q value of the next state obtained in step S4.8; S4.10, calculate the loss function, and update the Q network with the goal of minimizing the loss function; S4.11, determine τ%Γ target = 0, if so, update the target Q network based on the Q network parameters, that is, every training Γ target round, copy the parameters of the Q network to the target Q network once, and execute step S4.12, otherwise, directly execute step S4.12, where Γ target Indicates the update frequency of the target Q network; S4.12, determine τ%Ψ train = 0, if yes, the training is finished, otherwise, execute τ = τ + 1 and return to step S4.5, where Ψ train Represents the total number of training rounds.

5. The edge collaborative reasoning method of the diffusion model based on transfer reinforcement learning according to claim 4 is characterized in that: Before executing step S4.9, determine If yes, execute the diffusion step cycle update decision and then execute step S4.

9. Otherwise, execute step S4.9 directly, where: represents the diffusion step update period; The method for executing the diffusion step cycle update decision is: If yes, execute K=K-1 and update the diffusion step K. Otherwise, execute K=K+1 and update the diffusion step K. Indicates The average Q value within the diffusion step update cycle, Indicates The average Q value within the diffusion step update cycle, δ represents the Q value difference threshold; The expression of the updated diffusion step K is: In the formula, K min represents the preset minimum diffusion step, K max Indicates the preset maximum diffusion step.

6. The edge collaborative reasoning method of the diffusion model based on transfer reinforcement learning according to claim 3 is characterized in that: Before training the task offloading decision model, the fully connected layer of the task offloading decision model is randomly initialized and the parameters of other network layers are frozen.

7. The edge collaborative reasoning method of the diffusion model based on transfer reinforcement learning according to claim 1 is characterized in that: In step S5, the image generation model includes an image embedding layer, several Transformer blocks, an output processing layer and a quality discrimination module connected in sequence, and the quality discrimination module is also connected to the image embedding layer; the quality discrimination module uses a preset discrimination network to output the visual quality score of the intermediate image generated by the output processing layer, and compares the visual quality score with the visual quality threshold, and determines whether to output the intermediate image as the final image based on the comparison result.

8. A diffusion model edge collaborative reasoning device based on transfer reinforcement learning, comprising a plurality of user terminals and edge devices, characterized in that: The edge device includes a main node and several auxiliary nodes, and the main node is provided with: Data collection module: used to describe the offloading decision optimization problem of the image processing task received by the master node as a Markov decision process, wherein the Markov decision process includes a state space, an action space and a reward function; Decision model building module: used to build a task offloading decision model based on the DDQN network; Decision optimization module: used to train the task offloading decision model constructed by the model construction module based on the state space, action space and reward function obtained by the data collection module, generate the optimal offloading decision according to the trained task offloading decision model, and send the optimal offloading decision to the corresponding edge device and / or user terminal; Image processing module I: used to process the task using an image generation model based on the optimal offloading decision generated by the decision optimization module, wherein the image generation model is constructed based on the U-ViT model; The user terminal is provided with: Image processing module II: used to process the task using an image generation model based on the optimal offloading decision generated by the decision optimization module, wherein the image generation model is constructed based on the U-ViT model.

Images