Server-free edge computing function unloading method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of edge computing, and relates to a server-free edge computing function unloading method based on deep reinforcement learning, which comprises the following steps of: establishing a server-free edge computing network model which comprises a cloud center, an edge server, user equipment (UD) and an application program model; establishing an application program completion time model, a user equipment energy consumption model and a user cost model according to the server-free edge computing network model; establishing a server-free edge computing function unloading problem based on an application program completion time model, a user equipment energy consumption model and a user cost model; modeling a server-free edge computing function unloading problem as a Markov decision process, and solving the server-free edge computing function unloading problem by adopting deep reinforcement learning based on the Markov decision process to obtain a function unloading decision; according to the method, the problems of function deployment constraint and cold start delay are considered when the function unloading problem is constructed, and the effectiveness of function unloading decision can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of edge computing, and relates to a serverless edge computing function offloading method based on deep reinforcement learning. BACKGROUND

[0002] In recent years, the Internet of Things (IoT) technology has developed rapidly, which has enabled a large number of mobile devices to be interconnected through the Internet. However, the Internet of Things devices (such as sensors, smart phones, wearable devices, etc.) are generally limited in computing and energy resources. The traditional cloud computing framework alleviates this problem by offloading computing and caching tasks to the cloud. However, users need to bear the burden of resource supply and management: they need to pay for pre-allocated resources rather than actual usage, and need to manually configure load scaling strategies.

[0003] To solve the above problems, serverless computing (Serverless Computing) is proposed as a new cloud computing execution model. This architecture delegates infrastructure management (such as resource supply and scaling) to third-party providers, and its core Function as a Service (FaaS) allows application logic to be executed in a stateless function form in a containerized runtime environment on demand, without the need for pre-allocated resources, and is considered an important direction for the future development of cloud computing. At the same time, edge computing effectively solves the communication delay problem of cloud platforms caused by physical distance by deploying data processing locations of computing-intensive Internet of Things applications to the near-field domain within one hop of the terminal device. The industry is currently integrating serverless computing and edge computing to form a new paradigm of serverless edge computing (Serverless Edge Computing, SEC). This technology allows users to call edge resources without worrying about the runtime environment, resource allocation, load balancing, and scalability management, thereby significantly optimizing user experience and improving edge resource utilization.

[0004] Serverless edge computing is attracting attention due to its low latency and low cost advantages, and its core research focuses on architecture design, application deployment, and function offloading. Function offloading strategies have a significant impact on application performance (such as latency, terminal energy consumption, and user cost), and are a key research direction. Unlike traditional pure cloud environments, SEC needs to coordinate the heterogeneous resources of edge servers and cloud centers. Due to the limited edge resources, it is not possible to deploy all application functions and configure the execution environment, and the function container cold start latency needs to be considered additionally, resulting in a significant increase in offloading optimization complexity.

[0005] Existing researches mostly focus on single-objective optimization (e.g., only minimizing energy consumption or latency), but users often need to balance multiple objectives (e.g., prioritizing latency reduction in latency-sensitive scenarios and considering energy consumption and cost in budget-constrained scenarios) according to dynamic preferences. Single optimization methods are difficult to meet such needs, and multi-objective optimization solutions are urgently needed. In the field of multi-objective optimization, some researches apply multi-objective reinforcement learning (MORL) to edge computing task offloading, but these methods do not consider the function deployment constraints (e.g., edge servers can only deploy part of the functions) and cold start delay problems specific to SEC, resulting in ineffective handling of challenges caused by resource heterogeneity and dynamic preference changes in the SEC environment. In addition, these methods are prone to extrapolation errors when dealing with state distribution shifts, leading to poor model adaptability and inability to stably optimize multiple objectives in the dynamic environment of SEC. Therefore, a multi-objective function offloading method tailored for the SEC environment is particularly urgent. SUMMARY

[0006] To solve the above technical problems, the present application adopts a serverless edge computing function offloading method based on deep reinforcement learning, comprising:

[0007] S1, a serverless edge computing network model is established, including a cloud center, an edge server, a user device UD and an application program model;

[0008] S2, an application completion time model, a user device energy consumption model and a user cost model are established according to the serverless edge computing network model;

[0009] S3, a serverless edge computing function offloading problem is established based on the application completion time model, the user device energy consumption model and the user cost model;

[0010] S4, the serverless edge computing function offloading problem is modeled as a Markov decision process, and deep reinforcement learning is used to solve the serverless edge computing function offloading problem based on the Markov decision process to obtain a function offloading decision.

[0011] Advantages:

[0012] 1. The present application considers function deployment constraints and cold start delay problems when constructing the function offloading problem, which can better handle challenges caused by resource heterogeneity, thereby improving the effectiveness of function offloading decisions; 2. The present application generates preferences for each optimization objective through an LSTM network and adds the preferences for each optimization objective to a D3QN network to dynamically generate an optimal offloading strategy, achieving multi-objective utility maximization and improving the effectiveness of function offloading. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1A flow chart of a serverless edge computing function offloading method based on deep reinforcement learning is provided for an embodiment of the present application.

[0014] Figure 2 A system model schematic diagram is provided for an embodiment of the present application.

[0015] Figure 3 A flow chart of an LSTM-D3QN algorithm is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0017] As shown in the drawings, Figure 1 the present application adopts a serverless edge computing function offloading method based on deep reinforcement learning, which comprises:

[0018] S1, a serverless edge computing network model is established, including a cloud center, an edge server, a user device UD and an application program model; the application program function is deployed in the cloud center and the edge server;

[0019] The cloud center and the edge server cooperatively process the serverless application request of the user device UD, the application program is composed of application program functions that depend on each other, the function instance needs to be pre-deployed by the server, the cloud center can deploy all functions, and the edge server only deploys a subset of functions due to resource limitations. The application request of the UD is transmitted to the edge server through the wireless link in an uplink, if the target edge server does not deploy the function or the resource is insufficient, the execution time is set to infinity, and the execution result is returned to the UD through a downlink, because the data volume is small and the bandwidth is sufficient, the return time can be ignored.

[0020] The application program model is represented by a directed acyclic graph (DAG) ; wherein, represents a function set, and the function , , represents the input data size and the output data size of the function , , represents the memory size and the CPU cycle required for executing the function ; is a set of dependency relationships between functions, represents a function and function , in particular, function is the direct predecessor of function , function is the direct successor of function , , denote the set of direct predecessors and the set of direct successors of function respectively; before unloading function , all functions in must complete execution, denotes the function without direct predecessors, denotes the function without direct successors.

[0021] Cloud centers and edge servers are collectively referred to as a server set , these servers are heterogeneous, equipped with multi-core CPUs, whose clock frequency is defined as The set of functions deployed on all servers is denoted as , , denotes that all functions in are deployed on the servers of the cloud center, denotes that the edge servers deploy part of the functions in G.

[0022] Each server is provided with a container, and the container of each server corresponds to the deployed function one by one, serving as the execution unit of the corresponding function, and the memory resources allocated to the container are proportional to its computing power. The container of the edge server has three states: occupied, idle and available; among them, occupied means that the container is executing a function, idle means that the container has executed a function and the memory resources have not been recycled, and available means that the memory resources of the container have been recycled; when the function is executed, the container is in the occupied state, and after the function is executed, the container will be temporarily converted to the idle state to wait for the subsequent unloaded function, and if there is no unloading, the container will be converted to the available state to recycle the memory resources.

[0023] The resource state of the server to the function is represented by variables and ; among them, denotes that the server has an idle container of function , denotes that the server does not have an idle container of function , Indicates server There are sufficient available memory resources to execute the function. , Indicates server There are not enough available memory resources to execute the function. When the function Uninstall to At that time, if Can be executed immediately, if and If this happens, the available container needs to be restarted, resulting in a cold start time. .

[0024] S2. Based on the serverless edge computing network model, establish an application completion time model, a user device energy consumption model, and a user cost model;

[0025] Establishing an application completion time model includes:

[0026] The application's completion time is the total latency incurred during all function executions and data transfers, including function execution time and data transfer time. It is determined by the function... Input data transmission start and function Calculating the time difference between completions requires iteratively considering the processing time of each function in the application.

[0027] Specifically, the function The data transmission time includes: the transmission time of the input data. and set Maximum transmission time of output data for all functions in the program ;function Input data transmission time The specific representation is shown in equation (1):

[0028] (1)

[0029] in, Representation function The uninstallation strategy, if the function Unloaded to the server ,but equal ; The transmission rate between the user equipment (UD) and the edge server. , These represent the bandwidth and signal-to-noise ratio of the wireless channel between the user equipment (UD) and the edge server, respectively. Let be the transmission time from user equipment UD to the cloud center; to simplify the model, assume the transmission time from user equipment UD to the cloud center is... It is fixed. The value of will vary depending on the specific network environment (such as region, operator, time of day).

[0030] Assuming the transmission rate of the link between edge servers It is fixed. The value of needs to be determined based on the specific edge network deployment scenario (such as link type and bandwidth configuration) to fit the actual environment. set Transmission time of the output data of each function The specific representation is shown in equation (2):

[0031] (2)

[0032] in, , These are functions The unloading strategy and output data size, For function The direct predecessor.

[0033] function Data transmission time .

[0034] function Execution time The specific representation is shown in equation (3):

[0035] (3)

[0036] in, For execution function Required CPU cycles Cold start time,

[0037] In summary, the application completion time can be defined as... This indicates that the largest latest completion time is selected from all the latest completion times of functions without a direct successor as the application's completion time; where, Representation function The latest end time, , Representation function The earliest start time, The specific representation is shown in equation (4):

[0038] (4)

[0039] Establish a user equipment energy consumption model:

[0040] The user device UD energy consumption includes the energy consumption during data transmission and the energy consumption during idle time; the transmission energy consumption ; wherein, the transmission power of the pseudo user device UD; the idle energy consumption ; wherein, is the power of the user device UD when idle. Thus, the energy consumption can be defined as .

[0041] The user cost model is established including:

[0042] When the function is executed at the edge server or the cloud center, the user needs to pay according to the pricing scheme of the network provider. The AWS Lambda type pricing model (such as Lambda@Edge) is widely used in the industry, and the total cost of the user depends on: the number of function requests, the size of the memory allocated by the platform, and the execution time of the function.

[0043] The request cost of the function is denoted as , which is specifically represented as shown in equation (5):

[0044] (5)

[0045] wherein, is the number of requests of the function , and is the request cost coefficient, representing the cost per million requests;

[0046] Preferably, the request cost coefficient p of the cloud center is set to 0.2 dollars per million requests, and the request cost coefficient p of the edge server is set to 0.6 dollars per million requests.

[0047] The calculation cost of the function is denoted as , which is specifically represented as shown in equation (6):

[0048] (6)

[0049] wherein, represents the calculation cost coefficient, represents the actual execution memory allocated for the function .

[0050] Preferably, the calculation cost coefficient p of the cloud center is set to 0.00001667 dollars, and the calculation cost coefficient p of the edge server is set to 0.00005001 dollars.

[0051] According to the AWS Lambda rule, the platform allocates discrete memory in increments of 64MB, ranging from 128MB to 3008MB, The specific definition is shown in equation (7):

[0052] (7)

[0053] In summary, the processing function The total cost is denoted as , Therefore, the total user cost of completing a serverless application... Defined as K represents the number of functions in the application.

[0054] S3. Establish the serverless edge computing function offloading problem based on the application completion time model, user device energy consumption model, and user cost model;

[0055] For each function Select Uninstall Server To maximize the weighted utility of the three objectives (completion time, user cost, and user energy consumption) from the user's perspective, the serverless edge computing function offloading problem can be expressed as:

[0056] (8)

[0057] in, For a multi-objective optimization function, C1 represents the constraints of the function unloading decision, C2 represents the preference constraints of application completion time T, user device energy consumption E, and user device cost C, and C3 represents the dependency constraints between application functions. The function is specified. It can only be started after all its predecessor functions have been completed; L is the number of optimization targets. For the i-th function The decision to uninstall For preference vectors, The preference for the l-th optimization objective.

[0058] S4. The serverless edge computing function offloading problem is modeled as a Markov decision process. Based on the Markov decision process, deep reinforcement learning is used to solve the serverless edge computing function offloading problem and obtain the offloading decision.

[0059] Modeling the serverless edge computing function offloading problem as a Markov decision process includes:

[0060] Defined as a 4-tuple ,in, Representing the state space, Represents the action space, A vector representing the reward function. Represents a multi-objective preference space. Each element in is a preference vector The specific procedure includes:

[0061] Create state space: Use to represent the state space of SEC environment, The specific definition is represented as formula (9):

[0062] (9)

[0063] Where, represents the state at time , represents the function to be offloaded at time t, that is , represents the channel condition between user equipment UD and all servers at time t, is the channel condition between user equipment UD and server m, that is, the bandwidth and signal-to-noise ratio SNR of the channel, represents the computing power of the server at time t, that is, the clock frequency.

[0064] Create action space: offloading strategy of all stateless functions in serverless edge computing The specific definition is represented as formula (10):

[0065] (10)

[0066] Where, represents the action at time , is the offloading decision of the function to be offloaded at time ;

[0067] Create preference vector space :

[0068]

[0069] Where, is the preference vector at time t.

[0070] Create reward function:

[0071]

[0072]

[0073]

[0074]

[0075] in, , and It is a scalar reward for three optimization objectives: completion time, UD energy consumption, and user cost. Priority function sequence The former A subsequence composed of functions, a priority function sequence This is a function execution order generated based on the dependencies between functions, ensuring that each function is processed only after all its predecessor functions have completed. For sequence The completion time.

[0076] Solving the serverless edge computing function offloading problem using Markov decision processes and deep reinforcement learning includes:

[0077] S51. Initialize the LSTM-D3QN network, experience replay pool, and historical preference vector set; the LSTM-D3QN network includes: LSTM network and D3QN network;

[0078] The D3QN network includes: evaluation network and target network The evaluation network has the same structure as the target network, both including a feature extraction layer (which extracts the state). Mapping to high-dimensional features) and competitive value head (decomposing state value and action advantage).

[0079] The evaluation network employs a competitive architecture, decomposing state-action value into state value (intrinsic state value) and action advantage (relatively good or bad action), outputting an action value estimate for action selection and value calculation. The target network has the same structure as the evaluation network, with parameters that are a delayed, synchronized version of the evaluation network (updated every fixed number of steps), used to calculate a stable target value.

[0080] The experience replay area stores experience tuples containing the state, action, reward, and next state.

[0081] The historical preference vector set is used to store preference vectors that have already been trained.

[0082] S52. Obtaining state from the environment , will state Input the D3QN network to obtain the optimal action. Randomly select a preference vector from the preference vector space, and execute the optimal action based on the preference vector. Get the next state and rewards ; Constructing experience samples , to use empirical samples store the experience samples into an experience replay pool;

[0083] S53, repeat step S52 until the number of experience samples in the experience replay pool exceeds a preset batch size ;

[0084] S54, sample a batch of experience samples from the experience replay pool , randomly select a current preference vector from the preference vector space ;

[0085] Randomly selecting a current preference vector from the preference vector space ensures that the model can cover various priority combinations in the preference vector space, improving adaptability to dynamically changing preferences.

[0086] S54, train the LSTM-D3QN network according to the sampled batch of experience samples, the current preference vector , and the set of historical preference vectors, to obtain a trained LSTM-D3QN network; store the current preference vector in the set of historical preference vectors;

[0087] As the training progresses, the set of historical preference vectors will continue to expand, and new preference vectors may contain new distribution patterns (such as priority combinations that have never appeared before).

[0088] Training the LSTM-D3QN network includes:

[0089] S551, input the set of historical preference vectors into the LSTM network to obtain an optimal historical preference vector ;;

[0090] The LSTM model includes:

[0091] Input layer: receives the set of historical preference vectors, which includes multiple already trained preference vectors, each of which includes preferences for time, energy consumption, and cost;

[0092] LSTM layer: contains a number of hidden units (such as 64), which processes the preference vectors of the input layer through a gating mechanism (input gate, forget gate, output gate) to learn the temporal dependencies of the preference vectors, such as which preferences have appeared in similar scenarios or the pattern of preference changes over time;

[0093] Fully connected layer: maps the output of the LSTM layer to a relevance score vector , where is the score of the zth historical preference vector, and the historical preference vector with the highest score is the optimal historical preference vector.

[0094] The training objective of the LSTM model is to learn, through supervised learning, the LSTM to output a higher score for a preference vector in the history that is similar to the current preference For example, if the user has historically focused on cost, the LSTM is trained to output a higher score for a preference vector in the history that also focuses on cost.

[0095] S552, the sampled each experience sample is respectively concatenated with the historical preference vector and the current preference vector to obtain a composite experience sample of each experience sample , ;

[0096] Specifically, the state in the sampled each experience sample is respectively concatenated with the historical preference vector and the current preference vector to obtain a composite state and , and the composite experience sample of each experience sample is represented as , .

[0097] S553, the target Q value of each composite experience sample is calculated based on the evaluation network and the target network, and the current Q value of each composite experience sample is calculated based on the evaluation network;

[0098] The target Q value is calculated as

[0099]

[0100] wherein is the optimal action of the next state selected by the evaluation network, is the evaluation network, is an action in the action space, is the Q value (expected value of future reward) of the optimal action of the next state calculated by the target network, is the parameter of the target network, is the weight.

[0101] The current Q value is calculated as :

[0102]

[0103] ​

[0104] in, These are the outputs of the state value head and the advantage function head of the evaluation network, respectively.

[0105] S554, Based on historical preference vectors Current preference vector The loss function value is calculated for each composite empirical sample based on the target Q value and the current Q value. The parameters of the LSTM-D3QN network are updated based on the loss function value. When the loss function value is minimized (i.e. the loss function converges), the trained LSTM-D3QN network is obtained.

[0106] Mean squared error (MSE) loss is used to measure the target Q-value and the output of the LSTM network. The loss function is... The specific details are shown below:

[0107] (17)

[0108] in, For the composite empirical sample of the b-th empirical sample , The target Q value, , The composite empirical samples of the b-th empirical sample are respectively , Current Q value.

[0109] The network parameters are updated and evaluated by backpropagating the loss using a gradient descent algorithm (such as Adam). In the formula, It is the loss function with respect to parameters gradient, It is the learning rate.

[0110] In addition, the parameters of the target network Synchronization needs to be delayed; parameters should be copied from the evaluation network every C steps (e.g., C=1000). The purpose is to maintain the stability of the target network, avoid fluctuations in the network parameter evaluation causing oscillations in the calculation of the target Q value, and ensure training convergence.

[0111] S56. Obtain the function unloading decision based on the trained LSTM-D3QN network.

[0112] After training, the network is evaluated to determine if it has learned the state-action value function (the expected total reward for each action in each state). For any state... Optimal action The action with the largest Q value is represented as follows:

[0113]

[0114] This "state->action" mapping (optimal policy ) can offload stateless functions in serverless edge computing to edge servers, enabling dynamic and optimal decision-making.

[0115] The above examples further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made to the present application within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A serverless edge computing function offloading method based on deep reinforcement learning, characterized in that, include: S1. Establish a serverless edge computing network model, including cloud center, edge server, user device (UD), and application model; S2. Based on the serverless edge computing network model, establish an application completion time model, a user device energy consumption model, and a user cost model; S3. Establish the serverless edge computing function offloading problem based on the application completion time model, user device energy consumption model, and user cost model; S4. The serverless edge computing function offloading problem is modeled as a Markov decision process. Based on the Markov decision process, deep reinforcement learning is used to solve the serverless edge computing function offloading problem and obtain the function offloading decision.

2. The serverless edge computing function offloading method based on deep reinforcement learning according to claim 1, characterized in that, Application Model ;in, Represents a set of functions, functions K represents the number of functions in the application. It is a set of dependencies between functions. Representation function sum function The direct dependency between them , Representing functions respectively The size of the input data and the size of the output data. , These represent the functions to be executed. Required memory size and CPU cycles.

3. The serverless edge computing function offloading method based on deep reinforcement learning according to claim 1, characterized in that, Cloud centers and edge servers are collectively referred to as servers, and each server Multiple functions were deployed on each server. There are containers, each server Each container corresponds one-to-one with a deployed function, serving as the execution unit for that function. Containers have three states: occupied, idle, and available. Occupied indicates the container is executing a function; idle indicates the container has finished executing the function and its memory resources have not been reclaimed; and available indicates the container's memory resources have been reclaimed. (Server) For functions The resource status is represented as a variable. and ;in, Indicates server Existence function The free container Indicates server No function exists The free container Indicates server There are sufficient available memory resources to execute the function. , Indicates server There are not enough available memory resources to execute the function. .

4. The serverless edge computing function offloading method based on deep reinforcement learning according to claim 1, characterized in that, Establishing an application completion time model includes: ； ； ； ； Where T is the application's completion time. Representation function The latest end time, Representation function The earliest start time, Representation function The set of direct predecessor functions, This indicates a function that has no direct predecessor. This indicates a function that has no direct successor. For function Data transmission time, For function Execution time, For function The processing time is V, where V is the set of functions in the application.

5. The serverless edge computing function offloading method based on deep reinforcement learning according to claim 1, characterized in that, Establishing a user equipment cost model includes: ； ； ； Where C represents the user equipment cost. For function The cost of requesting For function The computational cost, where K represents the number of functions in the application. For function The number of requests, For the request cost coefficient, To calculate the cost coefficient, For function Execution time, Represented as a function The actual execution memory allocated depends on the function. Memory size required for execution Calculated.

6. The serverless edge computing function offloading method based on deep reinforcement learning according to claim 1, characterized in that, Establishing a user equipment energy consumption model includes: ； ； ； Where E represents the energy consumption of user equipment. For transmission energy consumption, For idle energy consumption, For the transmission power of user equipment UD, The power of user equipment UD when idle. For function The transmission time of the input data For function direct predecessor set The maximum transmission time of the output data of all functions in the program.

7. The serverless edge computing function offloading method based on deep reinforcement learning according to claim 1, characterized in that, The issue of unloading serverless edge computing functions is as follows: ； in, Let C1 be the constraint for the function offloading decision, C2 be the preference constraint for application completion time T, user device energy consumption E, and user device cost C, C3 be the dependency constraint between application functions, M be the total number of cloud center and edge servers, K be the number of application functions on the user device, and L be the number of optimization objectives. For the i-th function of the application The decision to uninstall For preference vectors, For the preference of the l-th optimization objective, For function The earliest start time, For function The latest end time, For function V is the set of direct predecessors of the application, and V is the set of functions of the application.

8. The serverless edge computing function offloading method based on deep reinforcement learning according to claim 7, characterized in that, Modeling the serverless edge computing function offloading problem as a Markov decision process includes: Create state space : ； in, Indicates time state, Indicates time The function to be unloaded. Indicates time Channel conditions between user equipment (UD) and all servers. This represents the computing power of all servers. Creating Action Space : ； in, Indicates time The action, For time Unloading decisions for functions to be unloaded; Create a preference vector space : ； in, Let be the preference vector at time t. , and These are the preferences for rewards based on application completion time, user device power consumption, and user device cost, respectively. Create a reward function: ； in, , and The rewards are for application completion time, user device power consumption, and user device cost, respectively.

9. The serverless edge computing function offloading method based on deep reinforcement learning according to claim 1, characterized in that, Solving the serverless edge computing function offloading problem includes: S51. Initialize the LSTM-D3QN network, experience replay pool, and historical preference vector set; the LSTM-D3QN network includes: LSTM network and D3QN network; S52. Obtaining state from the environment , will state Input the D3QN network to obtain the optimal action. Randomly select a preference vector from the preference vector space, and execute the optimal action based on the preference vector. Get the next state and rewards ; Constructing experience samples , to use empirical samples Store in the experience replay pool; S53. Repeat step S52 until the number of experience samples in the experience replay pool exceeds the preset batch size. ; S54. Sample a batch of experience samples from the experience replay pool, and randomly select a current preference vector from the preference vector space. ; S55. Based on the sampled batch of empirical samples and the current preference vector Train an LSTM-D3QN network using the historical preference vector set to obtain a trained LSTM-D3QN network; then use the current preference vector... Store in a historical preference vector set; S56. Obtain the function unloading decision based on the trained LSTM-D3QN network.

10. A serverless edge computing function offloading method based on deep reinforcement learning according to claim 9, characterized in that, The D3QN network includes: the evaluation network and the target network; training the LSTM-D3QN network includes: S551. Input the set of historical preference vectors into the LSTM network to obtain the optimal historical preference vector. ; S552, Each sampled empirical sample Respectively compared with historical preference vectors and current preference vector Feature concatenation is performed to obtain each empirical sample. Composite experience sample , ; S553. Calculate the target Q value for each composite experience sample based on the evaluation network and the target network, and calculate the current Q value for each composite experience sample based on the evaluation network. S554, Based on historical preference vectors Current preference vector The loss function value is calculated for each composite empirical sample based on the target Q value and the current Q value. The parameters of the LSTM-D3QN network are updated based on the loss function value. When the loss function value is minimized, the trained LSTM-D3QN network is obtained.

Images